[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:23 dist-test-slave-8tdl krb5kdc[28405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:23 dist-test-slave-8tdl krb5kdc[28405](info): set up 2 sockets
Aug 06 09:03:23 dist-test-slave-8tdl krb5kdc[28405](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 09:03:24.800081 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:24 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:25.033191 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:32989
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:25.523543 28418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:25.523955 28418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:25.524127 28418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:25.532271 28418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:25.532406 28418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:25.532500 28418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:25.532588 28418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:25.532652 28418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:25.544627 28418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:32989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:25.546816 28418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:25.548931 28418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:25.578543 28426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:25.578660 28424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:25.579066 28418 server_base.cc:1034] running on GCE node
W20260806 09:03:25.578655 28423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:25.580592 28418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:25.583194 28418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:25.584558 28418 hybrid_clock.cc:648] HybridClock initialized: now 1786007005584483 us; error 64 us; skew 500 ppm
I20260806 09:03:25.585214 28418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:25 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:25.589143 28418 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:25.592447 28418 webserver.cc:460] Webserver started at http://127.27.187.190:39911/ using document root <none> and password file <none>
I20260806 09:03:25.593578 28418 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:25.593704 28418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:25.594197 28418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:25.599035 28418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "d7be09b834704e21a3aa2fea2798038a"
format_stamp: "Formatted at 2026-08-06 09:03:25 on dist-test-slave-8tdl"
server_key: "e37d00fbbb630a24310aff6ead491e35"
server_key_iv: "a6846d465a2145666d7bdddcd5965465"
server_key_version: "encryptionkey@0"
I20260806 09:03:25.600243 28418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "d7be09b834704e21a3aa2fea2798038a"
format_stamp: "Formatted at 2026-08-06 09:03:25 on dist-test-slave-8tdl"
server_key: "e37d00fbbb630a24310aff6ead491e35"
server_key_iv: "a6846d465a2145666d7bdddcd5965465"
server_key_version: "encryptionkey@0"
I20260806 09:03:25.609318 28418 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20260806 09:03:25.616240 28434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:25.618604 28418 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260806 09:03:25.618990 28418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "d7be09b834704e21a3aa2fea2798038a"
format_stamp: "Formatted at 2026-08-06 09:03:25 on dist-test-slave-8tdl"
server_key: "e37d00fbbb630a24310aff6ead491e35"
server_key_iv: "a6846d465a2145666d7bdddcd5965465"
server_key_version: "encryptionkey@0"
I20260806 09:03:25.619220 28418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:25.637339 28418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:25.643877 28418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:25.644290 28418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:25.683754 28418 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:32989
I20260806 09:03:25.683784 28486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:32989 every 8 connection(s)
I20260806 09:03:25.686755 28418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:25.692524 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 28418
I20260806 09:03:25.693070 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:25.695003 28487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:25.695772 28398 external_mini_cluster.cc:1428] Setting key c9572ad19149200e1b20d5448763341f
I20260806 09:03:25.716677 28487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a: Bootstrap starting.
I20260806 09:03:25.724066 28487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:25.726577 28487 log.cc:824] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:25.738435 28487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a: No bootstrap required, opened a new log
Aug 06 09:03:25 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:25.751592 28487 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } }
I20260806 09:03:25.752537 28487 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:25.752861 28487 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7be09b834704e21a3aa2fea2798038a, State: Initialized, Role: FOLLOWER
I20260806 09:03:25.754496 28487 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } }
I20260806 09:03:25.755015 28487 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:25.755399 28487 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:25.755757 28487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:25.758661 28487 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } }
I20260806 09:03:25.760054 28487 leader_election.cc:302] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7be09b834704e21a3aa2fea2798038a; no voters:
I20260806 09:03:25.762024 28487 leader_election.cc:288] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:25.762707 28492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:25.765676 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:25.702087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.27.187.190:32989)
0806 09:03:25.704016 (+ 1929us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:25.704030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:25.705982 (+ 1952us) server_negotiation.cc:407] Connection header received
0806 09:03:25.718342 (+ 12360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:25.718370 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:25.719131 (+ 761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:25.719703 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:25.732472 (+ 12769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:25.735754 (+ 3282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:25.739539 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:25.740618 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:25.740890 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:25.749176 (+ 8286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:25.749238 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:25.749278 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:25.749370 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:25.754024 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:25.755467 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:25.755475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:25.755482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:25.755810 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:25.756810 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:25.756814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:25.756816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:25.757718 (+ 902us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:25.758258 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:25.759846 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0806 09:03:25.761005 (+ 1159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":916,"threads_started":1}
I20260806 09:03:25.767753 28492 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [term 1 LEADER]: Becoming Leader. State: Replica: d7be09b834704e21a3aa2fea2798038a, State: Running, Role: LEADER
I20260806 09:03:25.768570 28487 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:25.769034 28492 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } }
I20260806 09:03:25.778293 28493 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7be09b834704e21a3aa2fea2798038a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } } }
I20260806 09:03:25.778450 28494 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7be09b834704e21a3aa2fea2798038a. Latest consensus state: current_term: 1 leader_uuid: "d7be09b834704e21a3aa2fea2798038a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7be09b834704e21a3aa2fea2798038a" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 32989 } } }
I20260806 09:03:25.779099 28493 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:25.779175 28494 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:25.792538 28500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:25.806533 28500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:25.827632 28500 catalog_manager.cc:1347] Generated new cluster ID: 285a873e97a648f59197766543b1b59a
I20260806 09:03:25.828363 28500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:25.849835 28500 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:25.852254 28500 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:25.865123 28500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7be09b834704e21a3aa2fea2798038a: Generated new TSK 0
I20260806 09:03:25.866528 28500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:25.931154 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:26.365227 28513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:26.365644 28513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:26.365828 28513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:26.375309 28513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:26.375494 28513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:26.375695 28513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:26.388036 28513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:26.389601 28513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:26.391232 28513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:26.407970 28519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:26.408089 28518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:26.408071 28521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:26.409085 28513 server_base.cc:1034] running on GCE node
I20260806 09:03:26.409934 28513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:26.411245 28513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:26.412540 28513 hybrid_clock.cc:648] HybridClock initialized: now 1786007006412468 us; error 66 us; skew 500 ppm
I20260806 09:03:26.413012 28513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:26 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:26.417079 28513 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:26.419052 28513 webserver.cc:460] Webserver started at http://127.27.187.129:44977/ using document root <none> and password file <none>
I20260806 09:03:26.419921 28513 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:26.420039 28513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:26.420333 28513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:26.423882 28513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "e7d679b3f194411c977e0a82851a1050"
format_stamp: "Formatted at 2026-08-06 09:03:26 on dist-test-slave-8tdl"
server_key: "8c3353d3d67a14b9ca7c472fa7cc3d23"
server_key_iv: "035bfff20493b4780f0ae26331cd2c1b"
server_key_version: "encryptionkey@0"
I20260806 09:03:26.424818 28513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "e7d679b3f194411c977e0a82851a1050"
format_stamp: "Formatted at 2026-08-06 09:03:26 on dist-test-slave-8tdl"
server_key: "8c3353d3d67a14b9ca7c472fa7cc3d23"
server_key_iv: "035bfff20493b4780f0ae26331cd2c1b"
server_key_version: "encryptionkey@0"
I20260806 09:03:26.431167 28513 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:03:26.435626 28528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:26.437463 28513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:03:26.437631 28513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "e7d679b3f194411c977e0a82851a1050"
format_stamp: "Formatted at 2026-08-06 09:03:26 on dist-test-slave-8tdl"
server_key: "8c3353d3d67a14b9ca7c472fa7cc3d23"
server_key_iv: "035bfff20493b4780f0ae26331cd2c1b"
server_key_version: "encryptionkey@0"
I20260806 09:03:26.437935 28513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:26.459695 28513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:26.465691 28513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:26.465970 28513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:26.467177 28513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:26.469136 28513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:26.469230 28513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:26.469444 28513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:26.469502 28513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:26.541893 28513 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:41859
I20260806 09:03:26.541893 28643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:41859 every 8 connection(s)
I20260806 09:03:26.543843 28513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:26.548991 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 28513
I20260806 09:03:26.549254 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:26.550369 28398 external_mini_cluster.cc:1428] Setting key a61979f9fc503e93e0566d058de61709
Aug 06 09:03:26 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:03:26.577687 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:26.547949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:36205 (local address 127.27.187.190:32989)
0806 09:03:26.548853 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:26.548861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:26.550887 (+ 2026us) server_negotiation.cc:407] Connection header received
0806 09:03:26.554352 (+ 3465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:26.554356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:26.554432 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:26.554538 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:26.557844 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:26.559198 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:26.561967 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:26.562882 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:26.563092 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:26.568358 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:26.568368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:26.568377 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:26.568431 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:26.572640 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:26.574055 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:26.574061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:26.574063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:26.574319 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:26.575072 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:26.575076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:26.575079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:26.575549 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:26.575774 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:26.576917 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0806 09:03:26.577309 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":574,"threads_started":1}
I20260806 09:03:26.578838 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:26.549067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:32989 (local address 127.27.187.129:36205)
0806 09:03:26.550647 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:26.550732 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:26.553964 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:26.554960 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:26.554978 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:26.556283 (+ 1305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:26.557447 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:26.557479 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:26.559433 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:26.559442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:26.561665 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:26.561682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:26.563402 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:26.563413 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:26.563754 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:26.564640 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:26.564700 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:26.568064 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:26.572877 (+ 4813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:26.572890 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:26.572913 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:26.573832 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:26.574499 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:26.574507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:26.574512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:26.574898 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:26.575778 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:26.575787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:26.576359 (+ 572us) client_negotiation.cc:769] Sending connection context
0806 09:03:26.576900 (+ 541us) client_negotiation.cc:240] Negotiation successful
0806 09:03:26.577400 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":651,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:03:26.581246 28644 heartbeater.cc:347] Connected to a master server at 127.27.187.190:32989
I20260806 09:03:26.581784 28644 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:26.582864 28644 heartbeater.cc:511] Master 127.27.187.190:32989 requested a full tablet report, sending...
I20260806 09:03:26.587541 28451 ts_manager.cc:194] Registered new tserver with Master: e7d679b3f194411c977e0a82851a1050 (127.27.187.129:41859)
I20260806 09:03:26.589898 28451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:36205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:26.611665 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:27.057169 28650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:27.057488 28650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:27.057737 28650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:27.067245 28650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:27.067440 28650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:27.067657 28650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:27.080499 28650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:27.082021 28650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:27.083545 28650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:27.099495 28655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:27.099565 28656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:27.099530 28658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:27.100840 28650 server_base.cc:1034] running on GCE node
I20260806 09:03:27.101595 28650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:27.102815 28650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:27.104110 28650 hybrid_clock.cc:648] HybridClock initialized: now 1786007007104046 us; error 55 us; skew 500 ppm
I20260806 09:03:27.104528 28650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:27 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:27.109793 28650 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:27.112030 28650 webserver.cc:460] Webserver started at http://127.27.187.130:39043/ using document root <none> and password file <none>
I20260806 09:03:27.112957 28650 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:27.113149 28650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:27.113538 28650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:27.117038 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "50d74bfff1484e5688099db793278683"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "d1e7b8aeb527c54048d9b93f25f3a190"
server_key_iv: "63002effdaa9d8d77797e2032e1e2297"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.117954 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "50d74bfff1484e5688099db793278683"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "d1e7b8aeb527c54048d9b93f25f3a190"
server_key_iv: "63002effdaa9d8d77797e2032e1e2297"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.125908 28650 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260806 09:03:27.130398 28666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:27.132061 28650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:27.132362 28650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "50d74bfff1484e5688099db793278683"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "d1e7b8aeb527c54048d9b93f25f3a190"
server_key_iv: "63002effdaa9d8d77797e2032e1e2297"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.132607 28650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:27.168929 28650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:27.174175 28650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:27.174551 28650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:27.175822 28650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:27.177537 28650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:27.177630 28650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:27.177739 28650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:27.177812 28650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:27.227609 28650 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:40271
I20260806 09:03:27.227787 28786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:40271 every 8 connection(s)
I20260806 09:03:27.229353 28650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:27.238268 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 28650
I20260806 09:03:27.238512 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:27.239123 28398 external_mini_cluster.cc:1428] Setting key fbcd92849f0def6a62f393150fd98bba
Aug 06 09:03:27 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:27.256032 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:27.232554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:58767 (local address 127.27.187.190:32989)
0806 09:03:27.233060 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:27.233067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:27.234404 (+ 1337us) server_negotiation.cc:407] Connection header received
0806 09:03:27.237352 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:27.237358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:27.237456 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:27.237579 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:27.239758 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:27.240765 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:27.242647 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:27.243564 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:27.243733 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:27.248064 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:27.248070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:27.248073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:27.248110 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:27.252011 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:27.252964 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:27.252972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:27.252979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:27.253152 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:27.253616 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:27.253623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:27.253629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:27.254071 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:27.254232 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:27.255191 (+ 959us) server_negotiation.cc:299] Negotiation successful
0806 09:03:27.255498 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260806 09:03:27.256718 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:27.233214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:32989 (local address 127.27.187.130:58767)
0806 09:03:27.234191 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:27.234241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:27.236895 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:27.237928 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:27.237937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:27.238820 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:27.239529 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:27.239550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:27.240957 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:27.240962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:27.242490 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:27.242503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:27.243799 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:27.243807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:27.244140 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:27.244924 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:27.244949 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:27.247740 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:27.252252 (+ 4512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:27.252264 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:27.252287 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:27.252807 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:27.253280 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:27.253284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:27.253287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:27.253483 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:27.254264 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:27.254271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:27.254711 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 09:03:27.255028 (+ 317us) client_negotiation.cc:240] Negotiation successful
0806 09:03:27.255472 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":316,"threads_started":1}
I20260806 09:03:27.258453 28787 heartbeater.cc:347] Connected to a master server at 127.27.187.190:32989
I20260806 09:03:27.258910 28787 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:27.260088 28787 heartbeater.cc:511] Master 127.27.187.190:32989 requested a full tablet report, sending...
I20260806 09:03:27.262215 28451 ts_manager.cc:194] Registered new tserver with Master: 50d74bfff1484e5688099db793278683 (127.27.187.130:40271)
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
I20260806 09:03:27.263275 28451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:58767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:27.295697 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:03:27.594332 28644 heartbeater.cc:503] Master 127.27.187.190:32989 was elected leader, sending a full tablet report...
W20260806 09:03:27.819548 28794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:27.820003 28794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:27.820118 28794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:27.830981 28794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:27.831179 28794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:27.831372 28794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:27.844533 28794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:32989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:27.846024 28794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:27.847805 28794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:27.869776 28799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:27.869807 28800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:27.869863 28802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:27.871402 28794 server_base.cc:1034] running on GCE node
I20260806 09:03:27.872236 28794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:27.873472 28794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:27.874794 28794 hybrid_clock.cc:648] HybridClock initialized: now 1786007007874719 us; error 52 us; skew 500 ppm
I20260806 09:03:27.875301 28794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:27 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:27.880635 28794 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:27.883167 28794 webserver.cc:460] Webserver started at http://127.27.187.131:40185/ using document root <none> and password file <none>
I20260806 09:03:27.884095 28794 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:27.884356 28794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:27.884811 28794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:27.888955 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "0679d25cb1824cc2aaf77ffea0b0eeaa"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "12e4cf1718e6ead852f9ec7983db3735"
server_key_iv: "af5bc86f7bd381dd20b5e72e425033e0"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.890123 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "0679d25cb1824cc2aaf77ffea0b0eeaa"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "12e4cf1718e6ead852f9ec7983db3735"
server_key_iv: "af5bc86f7bd381dd20b5e72e425033e0"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.897838 28794 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20260806 09:03:27.903504 28809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:27.905246 28794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 09:03:27.905535 28794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "0679d25cb1824cc2aaf77ffea0b0eeaa"
format_stamp: "Formatted at 2026-08-06 09:03:27 on dist-test-slave-8tdl"
server_key: "12e4cf1718e6ead852f9ec7983db3735"
server_key_iv: "af5bc86f7bd381dd20b5e72e425033e0"
server_key_version: "encryptionkey@0"
I20260806 09:03:27.905824 28794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:27.926138 28794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:27.931530 28794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:27.931914 28794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:27.933310 28794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:27.935078 28794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:27.935307 28794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:27.935459 28794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:27.935537 28794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:28.019631 28794 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:35945
I20260806 09:03:28.019769 28926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:35945 every 8 connection(s)
I20260806 09:03:28.021476 28794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:28.024721 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 28794
I20260806 09:03:28.025019 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:28.025703 28398 external_mini_cluster.cc:1428] Setting key 38cee53d32ccc0f278d3c653a9f11d1f
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:28.056674 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.026210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:58377 (local address 127.27.187.190:32989)
0806 09:03:28.027236 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.027243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.029034 (+ 1791us) server_negotiation.cc:407] Connection header received
0806 09:03:28.032554 (+ 3520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.032559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.032706 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.032958 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.035295 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.036497 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.039344 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.040139 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.040338 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.046541 (+ 6203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.046547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.046551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.046590 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.051724 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.052929 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.052935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.052938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.053167 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.053909 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.053914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.053916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.054357 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.054532 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.055724 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.055932 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":608,"threads_started":1}
I20260806 09:03:28.058255 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.027074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:32989 (local address 127.27.187.131:58377)
0806 09:03:28.028815 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:28.028881 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:28.032154 (+ 3273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:28.033314 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:28.033325 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:28.033980 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:28.035046 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:28.035073 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.036806 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.036811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:28.039119 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:28.039139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.040577 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.040585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:28.040957 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.041950 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:28.041976 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:28.046219 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:28.051942 (+ 5723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:28.051957 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:28.051974 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:28.052752 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:28.053291 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:28.053295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:28.053297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:28.053699 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:28.054571 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:28.054651 (+ 80us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:28.055258 (+ 607us) client_negotiation.cc:769] Sending connection context
0806 09:03:28.055569 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 09:03:28.056139 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"thread_start_us":766,"threads_started":1}
I20260806 09:03:28.060774 28927 heartbeater.cc:347] Connected to a master server at 127.27.187.190:32989
I20260806 09:03:28.061252 28927 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:28.062474 28927 heartbeater.cc:511] Master 127.27.187.190:32989 requested a full tablet report, sending...
I20260806 09:03:28.064697 28451 ts_manager.cc:194] Registered new tserver with Master: 0679d25cb1824cc2aaf77ffea0b0eeaa (127.27.187.131:35945)
I20260806 09:03:28.065886 28451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:58377
I20260806 09:03:28.073807 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:28.142309 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.117534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.27.187.129:41859)
0806 09:03:28.119422 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.119429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.119459 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:03:28.119532 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.119539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.119802 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.119947 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.123427 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.124603 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.126656 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.127464 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.127586 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.133849 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.133867 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.133883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.133931 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.138369 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.139515 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.139523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.139526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.139763 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.140255 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.140258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.140260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.140733 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.141090 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.141435 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.141677 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":1387,"threads_started":1}
W20260806 09:03:28.144068 28557 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34382
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:28.202947 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.183248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.27.187.129:41859)
0806 09:03:28.183555 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.183565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.183979 (+ 414us) server_negotiation.cc:407] Connection header received
0806 09:03:28.184109 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.184114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.184193 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.184308 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.186030 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.187340 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.188858 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.189727 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.189866 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.195541 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.195548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.195552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.195599 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.199461 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.200541 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.200546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.200549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.200832 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.201342 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.201346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.201349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.201726 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.201961 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.202385 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.202591 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:03:28.205248 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 28513
I20260806 09:03:28.222254 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.206851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.27.187.129:41859)
0806 09:03:28.207123 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.207133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.207650 (+ 517us) server_negotiation.cc:407] Connection header received
0806 09:03:28.207721 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.207725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.207803 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.207904 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.209609 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.210367 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.212130 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.212628 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.212749 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.216081 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.216088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.216091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.216139 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.219516 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.220292 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.220296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.220298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.220431 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.220833 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.220836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.220838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.221129 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.221322 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.221693 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.221872 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:03:28.223368 28638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:28.266825 28787 heartbeater.cc:503] Master 127.27.187.190:32989 was elected leader, sending a full tablet report...
I20260806 09:03:28.324924 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 28513
I20260806 09:03:28.345710 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 28650
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:03:28.365406 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.346848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.27.187.130:40271)
0806 09:03:28.347521 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.347529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.347555 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:03:28.347625 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.347633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.347845 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.347999 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.349622 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.351098 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.352634 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.353466 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.353590 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.358488 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.358504 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.358520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.358568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.362111 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.363012 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.363020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.363022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.363160 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.363642 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.363646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.363648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.364129 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.364437 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.364761 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.365006 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
I20260806 09:03:28.366870 28782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:28.456314 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 28650
I20260806 09:03:28.476480 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 28794
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:03:28.496964 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:28.477650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.27.187.131:35945)
0806 09:03:28.478314 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:28.478322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:28.478355 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:03:28.478424 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:28.478431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:28.478658 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:28.478908 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:28.480462 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.481777 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.483564 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:28.484264 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:28.484393 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:28.489573 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:28.489598 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:28.489618 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:28.489665 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:28.493233 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.494217 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.494226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.494228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.494394 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:28.495019 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:28.495024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:28.495026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:28.495522 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:28.495814 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:28.496267 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 09:03:28.496573 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
I20260806 09:03:28.498530 28922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:28.586814 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 28794
I20260806 09:03:28.605666 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 28418
I20260806 09:03:28.607138 28481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:28.717697 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 28418
2026-08-06T09:03:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5162 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28949](info): set up 2 sockets
Aug 06 09:03:28 dist-test-slave-8tdl krb5kdc[28949](info): commencing operation
krb5kdc: starting...
W20260806 09:03:29.877959 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:30 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:30.116604 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38823
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:30.472345 28962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:30.472685 28962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:30.472856 28962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:30.480713 28962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:30.480846 28962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:30.480954 28962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:30.481057 28962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:30.481122 28962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:30.491698 28962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:30.493325 28962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:30.494824 28962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:30.509451 28967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:30.509461 28970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:30.509516 28968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:30.510798 28962 server_base.cc:1034] running on GCE node
I20260806 09:03:30.511783 28962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:30.513562 28962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:30.514822 28962 hybrid_clock.cc:648] HybridClock initialized: now 1786007010514769 us; error 40 us; skew 500 ppm
I20260806 09:03:30.515269 28962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:30 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007010, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:30.519004 28962 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:30.521173 28962 webserver.cc:460] Webserver started at http://127.27.187.190:45523/ using document root <none> and password file <none>
I20260806 09:03:30.522054 28962 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:30.522236 28962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:30.522637 28962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:30.526432 28962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "baa55eed5c824b3f8bed5559a76ad338"
format_stamp: "Formatted at 2026-08-06 09:03:30 on dist-test-slave-8tdl"
server_key: "aedadf528e14fcaacc844d62aa82f532"
server_key_iv: "79e0ed6449e279d461785367dbdfe0f7"
server_key_version: "encryptionkey@0"
I20260806 09:03:30.527467 28962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "baa55eed5c824b3f8bed5559a76ad338"
format_stamp: "Formatted at 2026-08-06 09:03:30 on dist-test-slave-8tdl"
server_key: "aedadf528e14fcaacc844d62aa82f532"
server_key_iv: "79e0ed6449e279d461785367dbdfe0f7"
server_key_version: "encryptionkey@0"
I20260806 09:03:30.534109 28962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:03:30.538540 28977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:30.540302 28962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:03:30.540552 28962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "baa55eed5c824b3f8bed5559a76ad338"
format_stamp: "Formatted at 2026-08-06 09:03:30 on dist-test-slave-8tdl"
server_key: "aedadf528e14fcaacc844d62aa82f532"
server_key_iv: "79e0ed6449e279d461785367dbdfe0f7"
server_key_version: "encryptionkey@0"
I20260806 09:03:30.540807 28962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:30.585980 28962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:30.591037 28962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:30.591374 28962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:30.612437 28962 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38823
I20260806 09:03:30.612449 29031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38823 every 8 connection(s)
I20260806 09:03:30.614187 28962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:30.618451 29032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:30.620694 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 28962
I20260806 09:03:30.620957 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:30.621804 28398 external_mini_cluster.cc:1428] Setting key 84f0f578a43ed680e6ae674880a8df18
I20260806 09:03:30.633016 29032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338: Bootstrap starting.
I20260806 09:03:30.637351 29032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:30.638906 29032 log.cc:824] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:03:30 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:30.642818 29032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338: No bootstrap required, opened a new log
I20260806 09:03:30.649945 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:30.624335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.27.187.190:38823)
0806 09:03:30.625278 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:30.625292 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:30.625341 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:03:30.627404 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:30.627432 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:30.627930 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:30.628320 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:30.630472 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:30.632274 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:30.634163 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:30.635017 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:30.635215 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:30.640789 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:30.640798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:30.640817 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:30.640860 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:30.644706 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:30.645923 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:30.645931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:30.645938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:30.646171 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:30.646620 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:30.646631 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:30.646639 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:30.647333 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:30.647658 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:30.648252 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 09:03:30.648743 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":458,"threads_started":1}
I20260806 09:03:30.652698 29032 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } }
I20260806 09:03:30.653201 29032 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:30.653326 29032 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa55eed5c824b3f8bed5559a76ad338, State: Initialized, Role: FOLLOWER
I20260806 09:03:30.653913 29032 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } }
I20260806 09:03:30.654142 29032 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:30.654245 29032 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:30.654367 29032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:30.656877 29032 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } }
I20260806 09:03:30.657500 29032 leader_election.cc:302] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa55eed5c824b3f8bed5559a76ad338; no voters:
I20260806 09:03:30.658252 29032 leader_election.cc:288] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:30.658377 29038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:30.659227 29038 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [term 1 LEADER]: Becoming Leader. State: Replica: baa55eed5c824b3f8bed5559a76ad338, State: Running, Role: LEADER
I20260806 09:03:30.659955 29038 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } }
I20260806 09:03:30.660953 29032 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:30.663298 29039 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa55eed5c824b3f8bed5559a76ad338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } } }
I20260806 09:03:30.663746 29039 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:30.664847 29040 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa55eed5c824b3f8bed5559a76ad338. Latest consensus state: current_term: 1 leader_uuid: "baa55eed5c824b3f8bed5559a76ad338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa55eed5c824b3f8bed5559a76ad338" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38823 } } }
I20260806 09:03:30.665422 29040 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:30.665568 29045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:30.671329 29045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:30.683279 29045 catalog_manager.cc:1347] Generated new cluster ID: cd75529ac0254928a15dba6e01e03bd0
I20260806 09:03:30.683439 29045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:30.702746 29045 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:30.704785 29045 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:30.729669 29045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa55eed5c824b3f8bed5559a76ad338: Generated new TSK 0
I20260806 09:03:30.730883 29045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:30.804315 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:31.175689 29059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:31.176088 29059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:31.176206 29059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:31.184111 29059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:31.184252 29059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:31.184377 29059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:31.194840 29059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:31.196209 29059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:31.197600 29059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:31.212445 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:31.212419 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:31.212399 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:31.213449 29059 server_base.cc:1034] running on GCE node
I20260806 09:03:31.214130 29059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:31.215319 29059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:31.216574 29059 hybrid_clock.cc:648] HybridClock initialized: now 1786007011216508 us; error 60 us; skew 500 ppm
I20260806 09:03:31.216943 29059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:31 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:31.220775 29059 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:31.222723 29059 webserver.cc:460] Webserver started at http://127.27.187.129:36539/ using document root <none> and password file <none>
I20260806 09:03:31.223542 29059 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:31.223711 29059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:31.224080 29059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:31.227562 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "9cffbb02e6cf4da4867b1c9a6241de4f"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "9e2309d69d495376235095deabc71d3e"
server_key_iv: "daa7d36302d0fe78c5b56533cb76ce82"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.228581 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "9cffbb02e6cf4da4867b1c9a6241de4f"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "9e2309d69d495376235095deabc71d3e"
server_key_iv: "daa7d36302d0fe78c5b56533cb76ce82"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.235105 29059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:03:31.239352 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.240899 29059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:03:31.241101 29059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "9cffbb02e6cf4da4867b1c9a6241de4f"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "9e2309d69d495376235095deabc71d3e"
server_key_iv: "daa7d36302d0fe78c5b56533cb76ce82"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.241401 29059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:31.259970 29059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:31.265101 29059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:31.265450 29059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:31.266537 29059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:31.268185 29059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:31.268313 29059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.268472 29059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:31.268548 29059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.308544 29059 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:37073
I20260806 09:03:31.308584 29190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:37073 every 8 connection(s)
I20260806 09:03:31.310153 29059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:31.320478 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29059
I20260806 09:03:31.320732 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:31.321542 28398 external_mini_cluster.cc:1428] Setting key b40923fcb763795c097abff481ed3714
Aug 06 09:03:31 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:31.336169 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:31.313196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:51413 (local address 127.27.187.190:38823)
0806 09:03:31.313699 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:31.313704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:31.314737 (+ 1033us) server_negotiation.cc:407] Connection header received
0806 09:03:31.316751 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:31.316757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:31.316858 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:31.316985 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:31.318667 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.319475 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.322198 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.322893 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.323026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:31.327451 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:31.327462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:31.327471 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:31.327529 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:31.331755 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:31.333085 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:31.333091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:31.333094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:31.333272 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:31.333943 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:31.333947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:31.333949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:31.334361 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:31.334479 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:31.335672 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0806 09:03:31.335884 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
I20260806 09:03:31.337909 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:31.313734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38823 (local address 127.27.187.129:51413)
0806 09:03:31.314498 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:31.314537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:31.316427 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:31.317253 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:31.317263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:31.317847 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:31.318466 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:31.318485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.319681 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.319686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:31.321933 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:31.321955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.323164 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.323172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:31.323498 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:31.324338 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:31.324363 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:31.327139 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:31.331996 (+ 4857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:31.332009 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:31.332034 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:31.332834 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:31.333479 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:31.333488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:31.333495 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:31.333800 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:31.334512 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:31.334524 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:31.335080 (+ 556us) client_negotiation.cc:769] Sending connection context
0806 09:03:31.335510 (+ 430us) client_negotiation.cc:240] Negotiation successful
0806 09:03:31.336171 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":296,"threads_started":1}
I20260806 09:03:31.340019 29191 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38823
I20260806 09:03:31.340415 29191 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:31.341361 29191 heartbeater.cc:511] Master 127.27.187.190:38823 requested a full tablet report, sending...
I20260806 09:03:31.343799 28996 ts_manager.cc:194] Registered new tserver with Master: 9cffbb02e6cf4da4867b1c9a6241de4f (127.27.187.129:37073)
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:03:31.345738 28996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:51413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:31.375298 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:31.751427 29197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:31.751840 29197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:31.752063 29197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:31.766914 29197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:31.767144 29197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:31.767429 29197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:31.789556 29197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:31.791188 29197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:31.792795 29197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:31.808302 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:31.809134 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:31.810086 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:31.811165 29197 server_base.cc:1034] running on GCE node
I20260806 09:03:31.811957 29197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:31.813333 29197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:31.814599 29197 hybrid_clock.cc:648] HybridClock initialized: now 1786007011814546 us; error 43 us; skew 500 ppm
I20260806 09:03:31.815016 29197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:31 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:31.818931 29197 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:31.820823 29197 webserver.cc:460] Webserver started at http://127.27.187.130:45843/ using document root <none> and password file <none>
I20260806 09:03:31.821631 29197 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:31.821795 29197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:31.822157 29197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:31.825848 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "ee673c265b69422c93bfc1aed31a38e0"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "655b0ef980450600815029daa5cbd88b"
server_key_iv: "47b829fce9f9f258c5061a387f12a491"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.827004 29197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "ee673c265b69422c93bfc1aed31a38e0"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "655b0ef980450600815029daa5cbd88b"
server_key_iv: "47b829fce9f9f258c5061a387f12a491"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.833935 29197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:03:31.838778 29213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.840723 29197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:03:31.840967 29197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ee673c265b69422c93bfc1aed31a38e0"
format_stamp: "Formatted at 2026-08-06 09:03:31 on dist-test-slave-8tdl"
server_key: "655b0ef980450600815029daa5cbd88b"
server_key_iv: "47b829fce9f9f258c5061a387f12a491"
server_key_version: "encryptionkey@0"
I20260806 09:03:31.841226 29197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:31.858003 29197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:31.863978 29197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:31.864365 29197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:31.865670 29197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:31.867470 29197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:31.867587 29197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.867771 29197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:31.867858 29197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:31.910140 29197 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:45879
I20260806 09:03:31.910247 29333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:45879 every 8 connection(s)
I20260806 09:03:31.911728 29197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:31.920420 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29197
I20260806 09:03:31.920670 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:31.921270 28398 external_mini_cluster.cc:1428] Setting key 4f7124d3aa6f2c2aab7a03f08fe1f2a1
Aug 06 09:03:31 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:31.942833 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:31.914870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:45205 (local address 127.27.187.190:38823)
0806 09:03:31.915540 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:31.915549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:31.916840 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 09:03:31.920808 (+ 3968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:31.920815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:31.920934 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:31.921075 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:31.924559 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.925901 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.929491 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.930332 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.930511 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:31.934864 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:31.934870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:31.934873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:31.934912 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:31.939056 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:31.940044 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:31.940049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:31.940052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:31.940197 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:31.940745 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:31.940749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:31.940751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:31.941076 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:31.941267 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:31.942078 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 09:03:31.942359 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":344,"threads_started":1}
I20260806 09:03:31.944173 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:31.915588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38823 (local address 127.27.187.130:45205)
0806 09:03:31.916541 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:31.916606 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:31.920309 (+ 3703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:31.921463 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:31.921482 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:31.922566 (+ 1084us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:31.924236 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:31.924283 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.926176 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.926186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:31.929207 (+ 3021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:31.929241 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:31.930652 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:31.930661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:31.930974 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:31.931943 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:31.931971 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:31.934505 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:31.939285 (+ 4780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:31.939302 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:31.939320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:31.939863 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:31.940352 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:31.940357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:31.940361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:31.940599 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:31.941223 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:31.941232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:31.941651 (+ 419us) client_negotiation.cc:769] Sending connection context
0806 09:03:31.941965 (+ 314us) client_negotiation.cc:240] Negotiation successful
0806 09:03:31.942643 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":334,"threads_started":1}
I20260806 09:03:31.946038 29334 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38823
I20260806 09:03:31.946489 29334 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:31.947501 29334 heartbeater.cc:511] Master 127.27.187.190:38823 requested a full tablet report, sending...
I20260806 09:03:31.949213 28996 ts_manager.cc:194] Registered new tserver with Master: ee673c265b69422c93bfc1aed31a38e0 (127.27.187.130:45879)
I20260806 09:03:31.950096 28996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:45205
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:31.987775 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:03:32.349583 29191 heartbeater.cc:503] Master 127.27.187.190:38823 was elected leader, sending a full tablet report...
W20260806 09:03:32.355477 29341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:32.355792 29341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:32.355890 29341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:32.363760 29341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:32.363898 29341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:32.364019 29341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:32.374600 29341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:32.376055 29341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:32.377496 29341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:32.393348 29347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:32.393352 29346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:32.393635 29349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:32.395022 29341 server_base.cc:1034] running on GCE node
I20260806 09:03:32.395720 29341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:32.396725 29341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:32.397955 29341 hybrid_clock.cc:648] HybridClock initialized: now 1786007012397904 us; error 35 us; skew 500 ppm
I20260806 09:03:32.398310 29341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:32.401973 29341 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:32.403869 29341 webserver.cc:460] Webserver started at http://127.27.187.131:37981/ using document root <none> and password file <none>
I20260806 09:03:32.404655 29341 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:32.404816 29341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:32.405249 29341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:32.408566 29341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9"
format_stamp: "Formatted at 2026-08-06 09:03:32 on dist-test-slave-8tdl"
server_key: "bf8c6794261ea2d262eb1b7161b48458"
server_key_iv: "25ca57b66cd9484a140e871b6a953329"
server_key_version: "encryptionkey@0"
I20260806 09:03:32.409461 29341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9"
format_stamp: "Formatted at 2026-08-06 09:03:32 on dist-test-slave-8tdl"
server_key: "bf8c6794261ea2d262eb1b7161b48458"
server_key_iv: "25ca57b66cd9484a140e871b6a953329"
server_key_version: "encryptionkey@0"
I20260806 09:03:32.415642 29341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 09:03:32.419910 29357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:32.421675 29341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:03:32.421900 29341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9"
format_stamp: "Formatted at 2026-08-06 09:03:32 on dist-test-slave-8tdl"
server_key: "bf8c6794261ea2d262eb1b7161b48458"
server_key_iv: "25ca57b66cd9484a140e871b6a953329"
server_key_version: "encryptionkey@0"
I20260806 09:03:32.422147 29341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:32.450837 29341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:32.456365 29341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:32.456784 29341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:32.458022 29341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:32.459676 29341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:32.459801 29341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:32.459985 29341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:32.460081 29341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:32.510344 29341 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:42943
I20260806 09:03:32.510604 29472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:42943 every 8 connection(s)
I20260806 09:03:32.512024 29341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:32.516240 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29341
I20260806 09:03:32.516691 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:32.517560 28398 external_mini_cluster.cc:1428] Setting key 95a64dbe0c3488f848c1315b4b9eae72
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:32.544402 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.516195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:47789 (local address 127.27.187.190:38823)
0806 09:03:32.517344 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.517353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.519164 (+ 1811us) server_negotiation.cc:407] Connection header received
0806 09:03:32.523286 (+ 4122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.523290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.523375 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.523471 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:32.525829 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.527124 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.529761 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.530368 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.530529 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.536566 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:32.536571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:32.536575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:32.536648 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:32.540511 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.541488 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.541494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.541498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.541643 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.542183 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.542189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.542192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.542526 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:32.542720 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.543526 (+ 806us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.543774 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":728,"threads_started":1}
I20260806 09:03:32.545712 29476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.517208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38823 (local address 127.27.187.131:47789)
0806 09:03:32.518963 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.519025 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.522966 (+ 3941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.523753 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.523763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.524647 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:32.525609 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.525632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.527411 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.527416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.529560 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.529577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.530840 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.530849 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.531157 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.532117 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:32.532145 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:32.536289 (+ 4144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:32.540711 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:32.540720 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:32.540736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:32.541342 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:32.541771 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:32.541782 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:32.541784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:32.542054 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:32.542711 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:32.542720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:32.543098 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.543393 (+ 295us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.544043 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"thread_start_us":970,"threads_started":1}
I20260806 09:03:32.548085 29473 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38823
I20260806 09:03:32.548521 29473 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:32.549743 29473 heartbeater.cc:511] Master 127.27.187.190:38823 requested a full tablet report, sending...
I20260806 09:03:32.551790 28996 ts_manager.cc:194] Registered new tserver with Master: a7046c0f4b2b4e8c9c4bc72a1f51dcb9 (127.27.187.131:42943)
I20260806 09:03:32.552863 28996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:47789
I20260806 09:03:32.562844 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:03:32.593500 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.572844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.27.187.190:38823)
0806 09:03:32.573050 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.573055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.574193 (+ 1138us) server_negotiation.cc:407] Connection header received
0806 09:03:32.574518 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.574523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.574747 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.574884 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:32.577828 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.579139 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.581014 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.581549 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.581719 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.585708 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:32.585715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:32.585719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:32.585761 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:32.589688 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.590575 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.590580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.590583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.590892 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.591469 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.591473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.591476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.591985 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:32.592315 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.592775 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.593010 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:03:32.603771 28996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:32.614579 28996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:32.664812 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.636544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42943 (local address 127.0.0.1:60158)
0806 09:03:32.640006 (+ 3462us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.640025 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.640160 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.651110 (+ 10950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.651123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.651179 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:32.652500 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.652527 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.656363 (+ 3836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.656372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.659900 (+ 3528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.659918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.663233 (+ 3315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.663258 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.663494 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.663571 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.663725 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.664017 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3246,"thread_start_us":2155,"threads_started":1}
I20260806 09:03:32.666368 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.636159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:37073 (local address 127.0.0.1:38252)
0806 09:03:32.638117 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.638172 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.638501 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.655363 (+ 16862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.655369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.655425 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:32.656240 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.656254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.659132 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.659138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.661922 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.661946 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.664971 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.664979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.665243 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.665269 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.665392 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.665665 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1504,"thread_start_us":3329,"threads_started":1}
I20260806 09:03:32.666471 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.636402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.27.187.129:37073)
0806 09:03:32.654551 (+ 18149us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.654570 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.654610 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:03:32.654751 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.654758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.655096 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.655691 (+ 595us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:32.656600 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.658933 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.662202 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.664603 (+ 2401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.664824 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.665087 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.665452 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.665720 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17835,"thread_start_us":17454,"threads_started":1}
I20260806 09:03:32.667197 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.637983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.27.187.131:42943)
0806 09:03:32.649637 (+ 11654us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.649650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.649704 (+ 54us) server_negotiation.cc:407] Connection header received
0806 09:03:32.649869 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.649882 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.650293 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.651989 (+ 1696us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:32.653247 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.656188 (+ 2941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.660102 (+ 3914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.662900 (+ 2798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.665806 (+ 2906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.666023 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.666135 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.666397 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11394,"thread_start_us":12613,"threads_started":1}
I20260806 09:03:32.669123 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.641081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:45879 (local address 127.0.0.1:43778)
0806 09:03:32.653729 (+ 12648us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.653759 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.653956 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.654857 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.654861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.654893 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:32.657266 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.657284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.659978 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.659983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.663874 (+ 3891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.663889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.668130 (+ 4241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.668138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.668332 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.668369 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.668509 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.668755 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12408,"thread_start_us":12073,"threads_started":1}
I20260806 09:03:32.669478 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.640397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.27.187.130:45879)
0806 09:03:32.649544 (+ 9147us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.649554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.654125 (+ 4571us) server_negotiation.cc:407] Connection header received
0806 09:03:32.654247 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.654254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.654593 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.654875 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:32.657665 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.659816 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.664119 (+ 4303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.667716 (+ 3597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.668001 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.668173 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.668627 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.668880 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8958,"thread_start_us":8752,"threads_started":1}
I20260806 09:03:32.672600 29406 tablet_service.cc:1467] Processing CreateTablet for tablet 6a06c7b9cae54aeab1ca5797faaae6d2 (DEFAULT_TABLE table=test-table [id=b6a4ff44296f468785ad1f289add2a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:32.672724 29124 tablet_service.cc:1467] Processing CreateTablet for tablet 6a06c7b9cae54aeab1ca5797faaae6d2 (DEFAULT_TABLE table=test-table [id=b6a4ff44296f468785ad1f289add2a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:32.674476 29262 tablet_service.cc:1467] Processing CreateTablet for tablet 6a06c7b9cae54aeab1ca5797faaae6d2 (DEFAULT_TABLE table=test-table [id=b6a4ff44296f468785ad1f289add2a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:32.675585 29406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a06c7b9cae54aeab1ca5797faaae6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:32.676527 29262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a06c7b9cae54aeab1ca5797faaae6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:32.677026 29124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a06c7b9cae54aeab1ca5797faaae6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:32.694866 29492 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Bootstrap starting.
I20260806 09:03:32.698503 29493 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Bootstrap starting.
I20260806 09:03:32.700682 29494 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Bootstrap starting.
I20260806 09:03:32.703114 29492 tablet_bootstrap.cc:654] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:32.704919 29493 tablet_bootstrap.cc:654] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:32.705564 29494 tablet_bootstrap.cc:654] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:32.705652 29492 log.cc:824] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:32.707779 29494 log.cc:824] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:32.707892 29493 log.cc:824] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:32.712859 29492 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: No bootstrap required, opened a new log
I20260806 09:03:32.713370 29492 ts_tablet_manager.cc:1397] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260806 09:03:32.713994 29494 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: No bootstrap required, opened a new log
I20260806 09:03:32.714372 29494 ts_tablet_manager.cc:1397] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260806 09:03:32.715754 29493 tablet_bootstrap.cc:492] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: No bootstrap required, opened a new log
I20260806 09:03:32.716269 29493 ts_tablet_manager.cc:1397] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260806 09:03:32.727170 29494 raft_consensus.cc:348] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.727881 29494 raft_consensus.cc:374] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:32.728111 29494 raft_consensus.cc:729] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee673c265b69422c93bfc1aed31a38e0, State: Initialized, Role: FOLLOWER
I20260806 09:03:32.728914 29494 consensus_queue.cc:260] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.731271 29334 heartbeater.cc:503] Master 127.27.187.190:38823 was elected leader, sending a full tablet report...
I20260806 09:03:32.731160 29492 raft_consensus.cc:348] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.732035 29492 raft_consensus.cc:374] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:32.732304 29492 raft_consensus.cc:729] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cffbb02e6cf4da4867b1c9a6241de4f, State: Initialized, Role: FOLLOWER
I20260806 09:03:32.732348 29494 ts_tablet_manager.cc:1428] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260806 09:03:32.733341 29492 consensus_queue.cc:260] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.733948 29493 raft_consensus.cc:348] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.734823 29493 raft_consensus.cc:374] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:32.734992 29493 raft_consensus.cc:729] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7046c0f4b2b4e8c9c4bc72a1f51dcb9, State: Initialized, Role: FOLLOWER
I20260806 09:03:32.736125 29493 consensus_queue.cc:260] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.736832 29492 ts_tablet_manager.cc:1428] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Time spent starting tablet: real 0.023s user 0.021s sys 0.003s
I20260806 09:03:32.738511 29473 heartbeater.cc:503] Master 127.27.187.190:38823 was elected leader, sending a full tablet report...
I20260806 09:03:32.740412 29493 ts_tablet_manager.cc:1428] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Time spent starting tablet: real 0.024s user 0.013s sys 0.010s
W20260806 09:03:32.765304 29474 tablet.cc:2369] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:32.783215 29500 raft_consensus.cc:482] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:32.783674 29500 raft_consensus.cc:504] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.786448 29500 leader_election.cc:288] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cffbb02e6cf4da4867b1c9a6241de4f (127.27.187.129:37073), ee673c265b69422c93bfc1aed31a38e0 (127.27.187.130:45879)
I20260806 09:03:32.795646 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.787400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:44195 (local address 127.27.187.129:37073)
0806 09:03:32.787695 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.787701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.787853 (+ 152us) server_negotiation.cc:407] Connection header received
0806 09:03:32.788103 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.788110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.788194 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.788322 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:32.789150 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.790511 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.793038 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.794260 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.794416 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.794480 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.795040 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.795180 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 09:03:32.795646 29476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.787473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:37073 (local address 127.27.187.131:44195)
0806 09:03:32.787728 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.787750 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.787923 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.788341 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.788345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.788468 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:32.788836 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.788858 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.790745 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.790751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.792805 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.792818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.794470 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.794478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.794860 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.794897 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.795006 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.795146 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 09:03:32.798223 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.788715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:52501 (local address 127.27.187.130:45879)
0806 09:03:32.789033 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.789039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.789589 (+ 550us) server_negotiation.cc:407] Connection header received
0806 09:03:32.789761 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.789768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.789849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.789947 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:32.791443 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.792530 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.794942 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.796672 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.796841 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.796897 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.797561 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.797709 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 09:03:32.797717 29144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a06c7b9cae54aeab1ca5797faaae6d2" candidate_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" is_pre_election: true
I20260806 09:03:32.798434 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.788017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:45879 (local address 127.27.187.131:52501)
0806 09:03:32.789454 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:32.789469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:32.789599 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:32.789979 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:32.789982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:32.790003 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:32.791102 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.791115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.792728 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.792739 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.794641 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:32.794663 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.797064 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.797072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:32.797340 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.797375 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:03:32.797505 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 09:03:32.797679 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":970,"threads_started":1}
I20260806 09:03:32.799116 29144 raft_consensus.cc:2454] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7046c0f4b2b4e8c9c4bc72a1f51dcb9 in term 0.
I20260806 09:03:32.800371 29359 leader_election.cc:302] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cffbb02e6cf4da4867b1c9a6241de4f, a7046c0f4b2b4e8c9c4bc72a1f51dcb9; no voters:
I20260806 09:03:32.800122 29283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a06c7b9cae54aeab1ca5797faaae6d2" candidate_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee673c265b69422c93bfc1aed31a38e0" is_pre_election: true
I20260806 09:03:32.800917 29500 raft_consensus.cc:2781] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:32.800904 29283 raft_consensus.cc:2454] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7046c0f4b2b4e8c9c4bc72a1f51dcb9 in term 0.
I20260806 09:03:32.801167 29500 raft_consensus.cc:482] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:32.801391 29500 raft_consensus.cc:3037] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:32.804844 29500 raft_consensus.cc:504] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.805714 29500 leader_election.cc:288] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [CANDIDATE]: Term 1 election: Requested vote from peers 9cffbb02e6cf4da4867b1c9a6241de4f (127.27.187.129:37073), ee673c265b69422c93bfc1aed31a38e0 (127.27.187.130:45879)
I20260806 09:03:32.806262 29144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a06c7b9cae54aeab1ca5797faaae6d2" candidate_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f"
I20260806 09:03:32.806284 29283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a06c7b9cae54aeab1ca5797faaae6d2" candidate_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee673c265b69422c93bfc1aed31a38e0"
I20260806 09:03:32.806540 29283 raft_consensus.cc:3037] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:32.806569 29144 raft_consensus.cc:3037] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:32.810067 29144 raft_consensus.cc:2454] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7046c0f4b2b4e8c9c4bc72a1f51dcb9 in term 1.
I20260806 09:03:32.810484 29283 raft_consensus.cc:2454] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7046c0f4b2b4e8c9c4bc72a1f51dcb9 in term 1.
I20260806 09:03:32.810945 29359 leader_election.cc:302] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cffbb02e6cf4da4867b1c9a6241de4f, a7046c0f4b2b4e8c9c4bc72a1f51dcb9; no voters:
I20260806 09:03:32.811545 29500 raft_consensus.cc:2781] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:32.812675 29500 raft_consensus.cc:686] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [term 1 LEADER]: Becoming Leader. State: Replica: a7046c0f4b2b4e8c9c4bc72a1f51dcb9, State: Running, Role: LEADER
W20260806 09:03:32.813253 29192 tablet.cc:2369] T 6a06c7b9cae54aeab1ca5797faaae6d2 P 9cffbb02e6cf4da4867b1c9a6241de4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:32.813561 29500 consensus_queue.cc:237] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } }
I20260806 09:03:32.820235 28994 catalog_manager.cc:5495] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7046c0f4b2b4e8c9c4bc72a1f51dcb9 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cffbb02e6cf4da4867b1c9a6241de4f" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 37073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7046c0f4b2b4e8c9c4bc72a1f51dcb9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee673c265b69422c93bfc1aed31a38e0" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45879 } health_report { overall_health: UNKNOWN } } }
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:32.863679 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.840295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.27.187.129:37073)
0806 09:03:32.840545 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.840550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.841662 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 09:03:32.841919 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.841923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.841995 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.842109 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:32.845014 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.846497 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.848516 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.849175 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.849334 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.855452 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:32.855471 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:32.855488 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:32.855536 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:32.859795 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.860714 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.860722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.860724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.860969 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.861433 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.861437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.861439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.862032 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:32.862523 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.862847 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.863123 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260806 09:03:32.915053 29335 tablet.cc:2369] T 6a06c7b9cae54aeab1ca5797faaae6d2 P ee673c265b69422c93bfc1aed31a38e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:32.928082 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.906151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.27.187.129:37073)
0806 09:03:32.906527 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.906534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.907104 (+ 570us) server_negotiation.cc:407] Connection header received
0806 09:03:32.907312 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.907316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.907402 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.907496 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:32.910167 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.911219 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.912915 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.913556 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.913726 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.919301 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:32.919307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:32.919310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:32.919351 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:32.924023 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.925079 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.925092 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.925095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.925348 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.926060 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.926067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.926076 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.926534 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:32.926975 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.927271 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.927567 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:03:32 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:32.991039 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.970103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.27.187.129:37073)
0806 09:03:32.970440 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.970450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:32.970896 (+ 446us) server_negotiation.cc:407] Connection header received
0806 09:03:32.971162 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:32.971166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:32.971239 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:32.971354 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:32.973051 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.974316 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.975919 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:32.976738 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:32.976896 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:32.983456 (+ 6560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:32.983462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:32.983466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:32.983511 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:32.987598 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.988656 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.988661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.988663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.988884 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:32.989358 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:32.989361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:32.989363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:32.989674 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:32.989873 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:32.990267 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 09:03:32.990489 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260806 09:03:32.993515 29104 tablet_service.cc:2908] Rejecting scan request for tablet 6a06c7b9cae54aeab1ca5797faaae6d2: Uninitialized: clean time has not yet been initialized
I20260806 09:03:32.998284 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29059
I20260806 09:03:33.016503 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:32.999554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.27.187.129:37073)
0806 09:03:32.999827 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:32.999838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:33.000331 (+ 493us) server_negotiation.cc:407] Connection header received
0806 09:03:33.000399 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:33.000403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:33.000485 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:33.000568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:33.002057 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.002894 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.004406 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.005449 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.005635 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:33.009463 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:33.009469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:33.009472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:33.009517 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:33.013022 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.013949 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.013953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.013956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.014141 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.014725 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.014734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.014736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.015191 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:33.015621 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:33.015770 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 09:03:33.016028 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:03:33.017576 29186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:33.116645 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29059
I20260806 09:03:33.137209 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29197
Aug 06 09:03:33 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:03:33.155714 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.138207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.27.187.130:45879)
0806 09:03:33.138447 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:33.138456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:33.138477 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:03:33.138547 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:33.138551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:33.138618 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:33.138767 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:33.140151 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.141102 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.142580 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.143261 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.143439 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:33.148540 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:33.148559 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:33.148577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:33.148627 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:33.152018 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.152990 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.152999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.153002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.153352 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.153868 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.153874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.153878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.154337 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:33.154669 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:33.155063 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 09:03:33.155344 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:03:33.156538 29329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:33.212990 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.212384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:37073 (local address 127.27.187.131:32865)
0806 09:03:33.212588 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:33.212807 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:37073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 09:03:33.213408 29359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.129:37073: connect: Connection refused (error 111)
I20260806 09:03:33.215286 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.214831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:37073 (local address 127.27.187.131:42541)
0806 09:03:33.215005 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:33.215101 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:37073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:03:33.215626 29359 consensus_peers.cc:487] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 -> Peer 9cffbb02e6cf4da4867b1c9a6241de4f (127.27.187.129:37073): Couldn't send request to peer 9cffbb02e6cf4da4867b1c9a6241de4f. Status: Network error: Client connection negotiation failed: client connection to 127.27.187.129:37073: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:03:33.252373 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29197
I20260806 09:03:33.271835 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29341
I20260806 09:03:33.276042 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.275531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:45879 (local address 127.27.187.131:56873)
0806 09:03:33.275734 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:33.275837 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.130:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 09:03:33.277324 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.276851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:45879 (local address 127.27.187.131:37365)
0806 09:03:33.277068 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:33.277161 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.130:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 09:03:33.277585 29359 consensus_peers.cc:487] T 6a06c7b9cae54aeab1ca5797faaae6d2 P a7046c0f4b2b4e8c9c4bc72a1f51dcb9 -> Peer ee673c265b69422c93bfc1aed31a38e0 (127.27.187.130:45879): Couldn't send request to peer ee673c265b69422c93bfc1aed31a38e0. Status: Network error: Client connection negotiation failed: client connection to 127.27.187.130:45879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 06 09:03:33 dist-test-slave-8tdl krb5kdc[28949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:03:33.291219 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:33.272943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.27.187.131:42943)
0806 09:03:33.273504 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:33.273509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:33.273528 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:03:33.273644 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:33.273656 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:33.273728 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:33.273833 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:33.275525 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.276625 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.278221 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:33.278660 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:33.278858 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:33.283943 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:33.283962 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:33.283979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:33.284041 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:33.287328 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.288260 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.288269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.288272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.288484 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:33.288984 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:33.288988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:33.288990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:33.289507 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:33.289958 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:33.290335 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 09:03:33.290662 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":314,"threads_started":1}
I20260806 09:03:33.292356 29468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:33.394888 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29341
I20260806 09:03:33.414803 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 28962
I20260806 09:03:33.416113 29027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:33.530009 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 28962
2026-08-06T09:03:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4811 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:33 dist-test-slave-8tdl krb5kdc[29524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:33 dist-test-slave-8tdl krb5kdc[29524](info): set up 2 sockets
Aug 06 09:03:33 dist-test-slave-8tdl krb5kdc[29524](info): commencing operation
krb5kdc: starting...
W20260806 09:03:34.677956 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:34 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:34.907866 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34471
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:35.276367 29538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:35.276696 29538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:35.276890 29538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:35.284420 29538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:35.284547 29538 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:03:35.284590 29538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:35.284650 29538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:35.284693 29538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:35.284756 29538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:35.295657 29538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34471
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:35.296998 29538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:35.298537 29538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:35.311837 29543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:35.312698 29544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:35.313014 29546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:35.314787 29538 server_base.cc:1034] running on GCE node
I20260806 09:03:35.315831 29538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:35.317432 29538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:35.318800 29538 hybrid_clock.cc:648] HybridClock initialized: now 1786007015318659 us; error 47 us; skew 500 ppm
I20260806 09:03:35.319219 29538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:35 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:35.326864 29538 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:35.329027 29538 webserver.cc:460] Webserver started at http://127.27.187.190:37503/ using document root <none> and password file <none>
I20260806 09:03:35.330104 29538 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:35.330346 29538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:35.330950 29538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:35.334612 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "cf75b40e57ec4e698392113a25bba3d5"
format_stamp: "Formatted at 2026-08-06 09:03:35 on dist-test-slave-8tdl"
server_key: "1847bc9831097db99b17ef4dcea250c3"
server_key_iv: "d92ae675c45ec831e80614aa96ae56d0"
server_key_version: "encryptionkey@0"
I20260806 09:03:35.335646 29538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "cf75b40e57ec4e698392113a25bba3d5"
format_stamp: "Formatted at 2026-08-06 09:03:35 on dist-test-slave-8tdl"
server_key: "1847bc9831097db99b17ef4dcea250c3"
server_key_iv: "d92ae675c45ec831e80614aa96ae56d0"
server_key_version: "encryptionkey@0"
I20260806 09:03:35.342097 29538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:03:35.346570 29553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:35.348318 29538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:35.348549 29538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "cf75b40e57ec4e698392113a25bba3d5"
format_stamp: "Formatted at 2026-08-06 09:03:35 on dist-test-slave-8tdl"
server_key: "1847bc9831097db99b17ef4dcea250c3"
server_key_iv: "d92ae675c45ec831e80614aa96ae56d0"
server_key_version: "encryptionkey@0"
I20260806 09:03:35.348821 29538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:35.368453 29538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:35.374153 29538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:35.374506 29538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:35.400064 29538 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34471
I20260806 09:03:35.400084 29617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34471 every 8 connection(s)
I20260806 09:03:35.401607 29538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:35.402294 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29538
I20260806 09:03:35.402558 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:35.403431 28398 external_mini_cluster.cc:1428] Setting key 326d96b21b235793b13dc567e4887ae9
I20260806 09:03:35.406512 29618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:03:35 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:35.421479 29618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Bootstrap starting.
I20260806 09:03:35.426373 29618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:35.428026 29618 log.cc:824] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:35.432323 29618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: No bootstrap required, opened a new log
I20260806 09:03:35.433641 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:35.405862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.27.187.190:34471)
0806 09:03:35.406760 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:35.406777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:35.406830 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:03:35.409272 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:35.409305 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:35.409872 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:35.410401 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:35.411999 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:35.414141 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:35.415882 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:35.417133 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:35.417414 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:35.422776 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:35.422802 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:35.422832 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:35.422904 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:35.427659 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:35.428761 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:35.428775 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:35.428796 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:35.429020 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:35.429612 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:35.429619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:35.429622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:35.430473 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:35.430873 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:35.431450 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 09:03:35.432011 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":362,"threads_started":1}
I20260806 09:03:35.442472 29618 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } }
I20260806 09:03:35.443009 29618 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:35.443107 29618 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf75b40e57ec4e698392113a25bba3d5, State: Initialized, Role: FOLLOWER
I20260806 09:03:35.443869 29618 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } }
I20260806 09:03:35.444111 29618 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:35.444227 29618 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:35.444383 29618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:35.446957 29618 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } }
I20260806 09:03:35.447695 29618 leader_election.cc:302] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf75b40e57ec4e698392113a25bba3d5; no voters:
I20260806 09:03:35.448385 29618 leader_election.cc:288] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:35.448534 29623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:35.449457 29623 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [term 1 LEADER]: Becoming Leader. State: Replica: cf75b40e57ec4e698392113a25bba3d5, State: Running, Role: LEADER
I20260806 09:03:35.450062 29623 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } }
I20260806 09:03:35.451030 29618 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:35.453150 29624 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf75b40e57ec4e698392113a25bba3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } } }
I20260806 09:03:35.453666 29624 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:35.455003 29625 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf75b40e57ec4e698392113a25bba3d5. Latest consensus state: current_term: 1 leader_uuid: "cf75b40e57ec4e698392113a25bba3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf75b40e57ec4e698392113a25bba3d5" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34471 } } }
I20260806 09:03:35.455397 29625 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:35.458719 29628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:35.465246 29628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:35.478050 29628 catalog_manager.cc:1347] Generated new cluster ID: 347458fb98e341539c1e36117fcfbe78
I20260806 09:03:35.478186 29628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:35.493146 29628 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:35.495281 29628 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:35.512063 29628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Generated new TSK 0
I20260806 09:03:35.513073 29628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:35.575773 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34471
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:35.952613 29644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:35.952997 29644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:35.953228 29644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:35.963037 29644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:35.963245 29644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:35.963411 29644 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:35.963564 29644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:35.975313 29644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:34471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:35.976811 29644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:35.978269 29644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:35.995065 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:35.995167 29652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:35.995091 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:35.997977 29644 server_base.cc:1034] running on GCE node
I20260806 09:03:35.998967 29644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:36.000257 29644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:36.001611 29644 hybrid_clock.cc:648] HybridClock initialized: now 1786007016001524 us; error 51 us; skew 500 ppm
I20260806 09:03:36.002302 29644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:36 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:36.008061 29644 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:36.010331 29644 webserver.cc:460] Webserver started at http://127.27.187.129:35477/ using document root <none> and password file <none>
I20260806 09:03:36.011288 29644 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:36.011409 29644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:36.011814 29644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:36.015424 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "01197aa2622446c68cf592a4048d621e"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "29246f02367d630a051eafd06666d1a4"
server_key_iv: "6e0bf160d21b7664de09081f7aa6b944"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.016489 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "01197aa2622446c68cf592a4048d621e"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "29246f02367d630a051eafd06666d1a4"
server_key_iv: "6e0bf160d21b7664de09081f7aa6b944"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.023186 29644 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:03:36.027532 29659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:36.029484 29644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:03:36.029707 29644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "01197aa2622446c68cf592a4048d621e"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "29246f02367d630a051eafd06666d1a4"
server_key_iv: "6e0bf160d21b7664de09081f7aa6b944"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.029963 29644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:36.050927 29644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:36.056525 29644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:36.056854 29644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:36.060685 29644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:36.060837 29644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:36.061003 29644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:36.061079 29644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:36 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:36.092973 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.061232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:52241 (local address 127.27.187.190:34471)
0806 09:03:36.064375 (+ 3143us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:36.064380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:36.064401 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:03:36.066996 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:36.067005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:36.067109 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:36.067267 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:36.070597 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.072028 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.075886 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.076494 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.076692 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.083597 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:36.083604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:36.083607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:36.083649 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:36.087947 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.089436 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.089442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.089446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.089612 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.090180 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.090186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.090189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.090509 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:36.090727 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:36.092212 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0806 09:03:36.092516 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3020,"thread_start_us":2905,"threads_started":1}
I20260806 09:03:36.094322 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.062016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.129:52241)
0806 09:03:36.063255 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:36.063316 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:36.066461 (+ 3145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:36.067630 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:36.067702 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:36.068670 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:36.070068 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.070102 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.072286 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.072300 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.075579 (+ 3279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.075620 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.077170 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.077184 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.077563 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.078818 (+ 1255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:36.078875 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:36.083246 (+ 4371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:36.088301 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.088319 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.088344 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.089248 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.089778 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.089785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.089794 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.090040 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.090659 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:36.090668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:36.091624 (+ 956us) client_negotiation.cc:769] Sending connection context
0806 09:03:36.092042 (+ 418us) client_negotiation.cc:240] Negotiation successful
0806 09:03:36.092850 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":548,"threads_started":1}
I20260806 09:03:36.127142 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.102060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:43435 (local address 127.27.187.190:34471)
0806 09:03:36.102351 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:36.102361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:36.103310 (+ 949us) server_negotiation.cc:407] Connection header received
0806 09:03:36.103733 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:36.103740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:36.103836 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:36.103966 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:36.105133 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.107020 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.110668 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.111798 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.112020 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.116831 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:36.116838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:36.116841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:36.116885 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:36.121943 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.124054 (+ 2111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.124063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.124069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.124351 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.125371 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.125376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.125378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.125782 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:36.125976 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:36.126466 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 09:03:36.126752 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:03:36.127367 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.102137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.129:43435)
0806 09:03:36.103138 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:36.103175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:36.103492 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:36.104002 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:36.104008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:36.104580 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:36.104924 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.104935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.107312 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.107323 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.110355 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.110394 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.112710 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.112725 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.113266 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.114504 (+ 1238us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:36.114525 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:36.116498 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:36.122323 (+ 5825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.122334 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.122341 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.123621 (+ 1280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.124576 (+ 955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.124657 (+ 81us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.124661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.124916 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.125932 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:36.125938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:36.126156 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 09:03:36.126318 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 09:03:36.126785 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":561,"threads_started":1}
I20260806 09:03:36.133654 29644 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:38951
I20260806 09:03:36.133749 29782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:38951 every 8 connection(s)
I20260806 09:03:36.135099 29644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:36.137953 29783 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34471
I20260806 09:03:36.138350 29783 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:36.139196 29783 heartbeater.cc:511] Master 127.27.187.190:34471 requested a full tablet report, sending...
I20260806 09:03:36.141700 29571 ts_manager.cc:194] Registered new tserver with Master: 01197aa2622446c68cf592a4048d621e (127.27.187.129:38951)
I20260806 09:03:36.143133 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29644
I20260806 09:03:36.143456 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:36.143599 29571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:52241
I20260806 09:03:36.144217 28398 external_mini_cluster.cc:1428] Setting key 030e45281c5749202f3485fa4c4cfb8e
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:36.200130 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34471
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:36.567022 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:36.567375 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:36.567553 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:36.575106 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:36.575241 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:36.575628 29787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:36.575814 29787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:36.587102 29787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:34471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:36.588698 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:36.590163 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:36.604796 29794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:36.604918 29797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:36.604918 29795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:36.605865 29787 server_base.cc:1034] running on GCE node
I20260806 09:03:36.606688 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:36.607854 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:36.609192 29787 hybrid_clock.cc:648] HybridClock initialized: now 1786007016609160 us; error 67 us; skew 500 ppm
I20260806 09:03:36.609546 29787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:36 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:36.613374 29787 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:36.615165 29787 webserver.cc:460] Webserver started at http://127.27.187.130:33289/ using document root <none> and password file <none>
I20260806 09:03:36.615942 29787 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:36.616082 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:36.616429 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:36.619724 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "ca9c10bab52044e2bd277765fc8d6c50"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "179164cf0be1fac8facdb465881565de"
server_key_iv: "3f3ebc00dc17a11efe1619c44062196b"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.620594 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "ca9c10bab52044e2bd277765fc8d6c50"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "179164cf0be1fac8facdb465881565de"
server_key_iv: "3f3ebc00dc17a11efe1619c44062196b"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.626487 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:03:36.630927 29804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:36.632519 29787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:36.632725 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ca9c10bab52044e2bd277765fc8d6c50"
format_stamp: "Formatted at 2026-08-06 09:03:36 on dist-test-slave-8tdl"
server_key: "179164cf0be1fac8facdb465881565de"
server_key_iv: "3f3ebc00dc17a11efe1619c44062196b"
server_key_version: "encryptionkey@0"
I20260806 09:03:36.632997 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:36.674424 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:36.679603 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:36.679883 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:36.682729 29787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:36.682819 29787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:36.682971 29787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:36.683053 29787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:36 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:36.714612 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.683675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:44013 (local address 127.27.187.190:34471)
0806 09:03:36.684286 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:36.684295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:36.685621 (+ 1326us) server_negotiation.cc:407] Connection header received
0806 09:03:36.688991 (+ 3370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:36.688999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:36.689115 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:36.689223 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:36.692731 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.693874 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.697550 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.698270 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.698435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.704037 (+ 5602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:36.704044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:36.704047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:36.704093 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:36.709339 (+ 5246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.710774 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.710785 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.710792 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.711013 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.711827 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.711835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.711842 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.712305 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:36.712498 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:36.713674 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0806 09:03:36.714078 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":326,"threads_started":1}
I20260806 09:03:36.716298 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.684444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.130:44013)
0806 09:03:36.685402 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:36.685466 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:36.688503 (+ 3037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:36.689961 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:36.689981 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:36.691094 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:36.692345 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.692403 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.694102 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.694110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.697316 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.697340 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.698891 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.698904 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.699255 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.700230 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:36.700317 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:36.703673 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:36.709705 (+ 6032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.709720 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.709745 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.710522 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.711260 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.711269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.711274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.711664 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.712461 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:36.712471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:36.713059 (+ 588us) client_negotiation.cc:769] Sending connection context
0806 09:03:36.713528 (+ 469us) client_negotiation.cc:240] Negotiation successful
0806 09:03:36.714346 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":355,"threads_started":1}
I20260806 09:03:36.741529 29787 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38543
I20260806 09:03:36.741540 29927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38543 every 8 connection(s)
I20260806 09:03:36.743590 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:36.746337 29928 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34471
I20260806 09:03:36.746656 29928 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:36.746943 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.725301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:37719 (local address 127.27.187.190:34471)
0806 09:03:36.725553 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:36.725562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:36.726601 (+ 1039us) server_negotiation.cc:407] Connection header received
0806 09:03:36.726879 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:36.726888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:36.726992 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:36.727128 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:36.729617 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.730801 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.733638 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.734471 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.734655 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.738811 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:36.738822 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:36.738827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:36.738887 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:36.743151 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.744332 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.744339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.744343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.744544 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:36.745152 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:36.745161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:36.745165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:36.745577 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:36.745768 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:36.746232 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 09:03:36.746500 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:03:36.747184 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:36.724911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.130:37719)
0806 09:03:36.726438 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:36.726466 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:36.726664 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:36.727535 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:36.727549 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:36.728573 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:36.729311 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.729335 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.730951 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.730963 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.733390 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:36.733418 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:36.734762 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:36.734771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:36.735202 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:36.736354 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:36.736389 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:36.738363 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:36.743499 (+ 5136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.743504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.743507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.744065 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.744746 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:36.744756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:36.744760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:36.744913 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:36.745725 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:36.745730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:36.745948 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 09:03:36.746113 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 09:03:36.746524 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":1230,"threads_started":1}
I20260806 09:03:36.747798 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29787
I20260806 09:03:36.748024 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:36.748591 28398 external_mini_cluster.cc:1428] Setting key 3dbb4ee521cbd0e2d0e79e4fa23f4ff4
I20260806 09:03:36.749543 29928 heartbeater.cc:511] Master 127.27.187.190:34471 requested a full tablet report, sending...
I20260806 09:03:36.751427 29570 ts_manager.cc:194] Registered new tserver with Master: ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543)
I20260806 09:03:36.752794 29570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:44013
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:36.803578 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34471
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:03:37.147486 29783 heartbeater.cc:503] Master 127.27.187.190:34471 was elected leader, sending a full tablet report...
W20260806 09:03:37.157788 29932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:37.158497 29932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:37.158799 29932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:37.167282 29932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:37.167464 29932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:37.167625 29932 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:37.167768 29932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:37.178450 29932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:34471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:37.179965 29932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:37.181478 29932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:37.196931 29938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:37.197027 29940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:37.196955 29937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:37.198185 29932 server_base.cc:1034] running on GCE node
I20260806 09:03:37.199111 29932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:37.200394 29932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:37.201731 29932 hybrid_clock.cc:648] HybridClock initialized: now 1786007017201640 us; error 72 us; skew 500 ppm
I20260806 09:03:37.202181 29932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:37 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:37.206645 29932 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:37.208685 29932 webserver.cc:460] Webserver started at http://127.27.187.131:45423/ using document root <none> and password file <none>
I20260806 09:03:37.209607 29932 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:37.209733 29932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:37.210203 29932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:37.213590 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "b966a98ea843438eaad46670378924e9"
format_stamp: "Formatted at 2026-08-06 09:03:37 on dist-test-slave-8tdl"
server_key: "a99227e59436f1a62a1540c08d07c52a"
server_key_iv: "918bacaba6a4fa04ca3d64b7e44e6961"
server_key_version: "encryptionkey@0"
I20260806 09:03:37.214521 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "b966a98ea843438eaad46670378924e9"
format_stamp: "Formatted at 2026-08-06 09:03:37 on dist-test-slave-8tdl"
server_key: "a99227e59436f1a62a1540c08d07c52a"
server_key_iv: "918bacaba6a4fa04ca3d64b7e44e6961"
server_key_version: "encryptionkey@0"
I20260806 09:03:37.221952 29932 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:03:37.226328 29948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:37.227913 29932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:03:37.228111 29932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "b966a98ea843438eaad46670378924e9"
format_stamp: "Formatted at 2026-08-06 09:03:37 on dist-test-slave-8tdl"
server_key: "a99227e59436f1a62a1540c08d07c52a"
server_key_iv: "918bacaba6a4fa04ca3d64b7e44e6961"
server_key_version: "encryptionkey@0"
I20260806 09:03:37.228401 29932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:37.245172 29932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:37.250967 29932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:37.251304 29932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:37.254920 29932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:37.255040 29932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:37.255172 29932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:37.255249 29932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:37 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:37.288046 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.255726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:45405 (local address 127.27.187.190:34471)
0806 09:03:37.256345 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.256351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.257740 (+ 1389us) server_negotiation.cc:407] Connection header received
0806 09:03:37.261154 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.261164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.261295 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.261434 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:37.264515 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.265924 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.269737 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.270436 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.270636 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.276972 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:37.276984 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:37.276992 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:37.277057 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:37.282909 (+ 5852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.284353 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.284364 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.284371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.284754 (+ 383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.285593 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.285602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.285607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.285999 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:37.286182 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.287340 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.287629 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":321,"threads_started":1}
I20260806 09:03:37.289201 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.256448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.131:45405)
0806 09:03:37.257455 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.257504 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.260627 (+ 3123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.261847 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.261861 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.263008 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:37.264196 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.264246 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.266205 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.266215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.269369 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.269395 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.270892 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.270905 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.271347 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.272472 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:37.272524 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:37.276542 (+ 4018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:37.283307 (+ 6765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.283334 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.283363 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.284108 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.284949 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.284956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.284962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.285348 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.286170 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:37.286179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:37.286814 (+ 635us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.287198 (+ 384us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.287917 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":430,"threads_started":1}
I20260806 09:03:37.312825 29932 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:37981
I20260806 09:03:37.312826 30070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:37981 every 8 connection(s)
I20260806 09:03:37.314317 29932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:37.315308 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.297584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:47105 (local address 127.27.187.190:34471)
0806 09:03:37.297975 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.297984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.298308 (+ 324us) server_negotiation.cc:407] Connection header received
0806 09:03:37.298566 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.298572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.298670 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.298816 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:37.300477 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.301300 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.303604 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.304682 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.304884 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.307981 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:37.307990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:37.307994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:37.308044 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:37.312129 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.313020 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.313027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.313031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.313285 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.313697 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.313703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.313706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.314016 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:37.314186 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.314631 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.314928 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260806 09:03:37.315462 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.297489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.27.187.131:47105)
0806 09:03:37.298192 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.298213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.298422 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.299004 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.299010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.299743 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:37.300280 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.300300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.301456 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.301465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.303341 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.303363 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.304920 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.304925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.305042 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.305919 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:37.305939 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:37.307510 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:37.312313 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.312325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.312328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.312872 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.313409 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.313415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.313425 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.313563 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.314141 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:37.314145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:37.314344 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.314536 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.314962 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":460,"threads_started":1}
I20260806 09:03:37.317521 30071 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34471
I20260806 09:03:37.317893 30071 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:37.318941 30071 heartbeater.cc:511] Master 127.27.187.190:34471 requested a full tablet report, sending...
I20260806 09:03:37.320633 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 29932
I20260806 09:03:37.320660 29571 ts_manager.cc:194] Registered new tserver with Master: b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:37.320813 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:37.321286 28398 external_mini_cluster.cc:1428] Setting key 83b80dcfbe1cdb8c003f6aeaa72def00
I20260806 09:03:37.321693 29571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:45405
I20260806 09:03:37.324769 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:03:37 dist-test-slave-8tdl krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:03:37 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:37.382488 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.366034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.27.187.190:34471)
0806 09:03:37.366278 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.366287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.366872 (+ 585us) server_negotiation.cc:407] Connection header received
0806 09:03:37.367039 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.367043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.367128 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.367230 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:37.368876 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.369615 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.371393 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.371854 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.371964 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.376394 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:37.376400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:37.376403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:37.376448 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:37.379476 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.380364 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.380369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.380372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.380524 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.380991 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.380995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.380998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.381375 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:37.381670 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.381998 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.382178 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:03:37.387352 29571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:37.395088 29571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:37.428764 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.410920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38951 (local address 127.0.0.1:49462)
0806 09:03:37.411896 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.411954 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.412244 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.418531 (+ 6287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.418544 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.418591 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.419185 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.419212 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.420790 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.420799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.424022 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.424051 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.427477 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.427486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.427869 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.427950 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.428103 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.428351 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":353,"threads_started":1}
I20260806 09:03:37.430205 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.411352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.27.187.129:38951)
0806 09:03:37.417541 (+ 6189us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.417548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.417577 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:03:37.417823 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.417837 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.418155 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.418485 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.419428 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.420630 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.424358 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.426613 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.426891 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.427168 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.429614 (+ 2446us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.429850 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5979,"thread_start_us":5718,"threads_started":1}
I20260806 09:03:37.435946 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.412930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38543 (local address 127.0.0.1:34892)
0806 09:03:37.416453 (+ 3523us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.416483 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.416744 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.424747 (+ 8003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.424754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.424791 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.427637 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.427659 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.429868 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.429874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.432495 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.432523 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.434999 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.435023 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.435218 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.435254 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.435376 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.435542 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3267,"mutex_wait_us":2,"thread_start_us":11407,"threads_started":1}
I20260806 09:03:37.436358 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.413891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.27.187.131:37981)
0806 09:03:37.415674 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.415702 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.424820 (+ 9118us) server_negotiation.cc:407] Connection header received
0806 09:03:37.425166 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.425177 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.425523 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.425706 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.428053 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.429776 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.432870 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.434494 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.434717 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.434911 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.435761 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.435959 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1482,"thread_start_us":1182,"threads_started":1}
I20260806 09:03:37.435322 29712 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0134c198549d89231258c3367a739 (DEFAULT_TABLE table=test-table [id=4097e099ec914088a34e7283db9025e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.436791 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.413505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.27.187.130:38543)
0806 09:03:37.423244 (+ 9739us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.423254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.423289 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:03:37.423405 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.423415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.423879 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.427931 (+ 4052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.428213 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.429698 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.432899 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.434697 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.435787 (+ 1090us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.435974 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.436197 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.436545 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9526,"thread_start_us":9369,"threads_started":1}
I20260806 09:03:37.437060 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.413305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:37981 (local address 127.0.0.1:35446)
0806 09:03:37.424654 (+ 11349us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.424687 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.424956 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.426068 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.426073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.426101 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.427746 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.427771 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.429977 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.429983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.432654 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.432670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.435137 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.435143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.435491 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.435543 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.436506 (+ 963us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.436673 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11201,"mutex_wait_us":81,"thread_start_us":2638,"threads_started":1}
I20260806 09:03:37.437682 29712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e0134c198549d89231258c3367a739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.440624 30000 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0134c198549d89231258c3367a739 (DEFAULT_TABLE table=test-table [id=4097e099ec914088a34e7283db9025e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.441397 29856 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0134c198549d89231258c3367a739 (DEFAULT_TABLE table=test-table [id=4097e099ec914088a34e7283db9025e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.442804 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e0134c198549d89231258c3367a739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.443280 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e0134c198549d89231258c3367a739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.456375 30091 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Bootstrap starting.
I20260806 09:03:37.458945 30092 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Bootstrap starting.
I20260806 09:03:37.460757 30091 tablet_bootstrap.cc:654] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.461592 30093 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Bootstrap starting.
I20260806 09:03:37.463729 30092 tablet_bootstrap.cc:654] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.463730 30091 log.cc:824] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:37.465396 30092 log.cc:824] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:37.465487 30093 tablet_bootstrap.cc:654] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.467782 30093 log.cc:824] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:37.468317 30091 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: No bootstrap required, opened a new log
I20260806 09:03:37.468741 30091 ts_tablet_manager.cc:1397] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260806 09:03:37.470180 30092 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: No bootstrap required, opened a new log
I20260806 09:03:37.470538 30092 ts_tablet_manager.cc:1397] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260806 09:03:37.479373 30093 tablet_bootstrap.cc:492] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: No bootstrap required, opened a new log
I20260806 09:03:37.479835 30093 ts_tablet_manager.cc:1397] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260806 09:03:37.487008 30091 raft_consensus.cc:348] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.487172 30092 raft_consensus.cc:348] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.487833 30091 raft_consensus.cc:374] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.487829 30092 raft_consensus.cc:374] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.488018 30092 raft_consensus.cc:729] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01197aa2622446c68cf592a4048d621e, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.488046 30091 raft_consensus.cc:729] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b966a98ea843438eaad46670378924e9, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.488969 30091 consensus_queue.cc:260] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.489055 30092 consensus_queue.cc:260] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.490971 30071 heartbeater.cc:503] Master 127.27.187.190:34471 was elected leader, sending a full tablet report...
I20260806 09:03:37.491768 30091 ts_tablet_manager.cc:1428] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Time spent starting tablet: real 0.023s user 0.012s sys 0.009s
I20260806 09:03:37.493177 30093 raft_consensus.cc:348] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.493780 30093 raft_consensus.cc:374] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.493903 30093 raft_consensus.cc:729] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9c10bab52044e2bd277765fc8d6c50, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.494604 30092 ts_tablet_manager.cc:1428] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s
I20260806 09:03:37.494606 30093 consensus_queue.cc:260] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.496353 29928 heartbeater.cc:503] Master 127.27.187.190:34471 was elected leader, sending a full tablet report...
I20260806 09:03:37.497268 30093 ts_tablet_manager.cc:1428] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260806 09:03:37.540205 30099 raft_consensus.cc:482] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:37.540895 30099 raft_consensus.cc:504] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.543426 30099 leader_election.cc:288] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01197aa2622446c68cf592a4048d621e (127.27.187.129:38951), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:37.553779 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.544224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:57967 (local address 127.27.187.129:38951)
0806 09:03:37.544463 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.544468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.545221 (+ 753us) server_negotiation.cc:407] Connection header received
0806 09:03:37.545425 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.545429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.545495 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.545685 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.547046 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.548056 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.551111 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.552463 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.552701 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.552833 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.553306 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.553450 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:03:37.554111 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.545766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:56869 (local address 127.27.187.131:37981)
0806 09:03:37.545963 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.545968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.546367 (+ 399us) server_negotiation.cc:407] Connection header received
0806 09:03:37.546659 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.546663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.546755 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.546865 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.547785 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.548945 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.551597 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.553018 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.553254 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.553336 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.553634 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.553835 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:03:37.554234 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.545292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:37981 (local address 127.27.187.130:56869)
0806 09:03:37.546156 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.546177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.546443 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.546922 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.546927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.546955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.547580 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.547591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.549171 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.549177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.551409 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.551422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.553288 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.553295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.553457 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.553490 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.553586 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.553740 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":20,"thread_start_us":392,"threads_started":1}
I20260806 09:03:37.554279 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.544014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38951 (local address 127.27.187.130:57967)
0806 09:03:37.545005 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.545058 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.545226 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.545902 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.545907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.545941 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.546905 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.546921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.548242 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.548248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.550899 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.550918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.552768 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.552778 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.553014 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.553062 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.553283 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.553447 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":29,"thread_start_us":568,"threads_started":1}
I20260806 09:03:37.555831 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e0134c198549d89231258c3367a739" candidate_uuid: "ca9c10bab52044e2bd277765fc8d6c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9" is_pre_election: true
I20260806 09:03:37.555883 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e0134c198549d89231258c3367a739" candidate_uuid: "ca9c10bab52044e2bd277765fc8d6c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01197aa2622446c68cf592a4048d621e" is_pre_election: true
I20260806 09:03:37.556402 30020 raft_consensus.cc:2454] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9c10bab52044e2bd277765fc8d6c50 in term 0.
I20260806 09:03:37.556402 29732 raft_consensus.cc:2454] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9c10bab52044e2bd277765fc8d6c50 in term 0.
I20260806 09:03:37.557224 29807 leader_election.cc:302] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, ca9c10bab52044e2bd277765fc8d6c50; no voters:
I20260806 09:03:37.557777 30099 raft_consensus.cc:2781] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:37.557993 30099 raft_consensus.cc:482] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:37.558140 30099 raft_consensus.cc:3037] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.560828 30099 raft_consensus.cc:504] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.561491 30099 leader_election.cc:288] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [CANDIDATE]: Term 1 election: Requested vote from peers 01197aa2622446c68cf592a4048d621e (127.27.187.129:38951), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:37.562136 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e0134c198549d89231258c3367a739" candidate_uuid: "ca9c10bab52044e2bd277765fc8d6c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01197aa2622446c68cf592a4048d621e"
I20260806 09:03:37.562449 29732 raft_consensus.cc:3037] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.562475 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e0134c198549d89231258c3367a739" candidate_uuid: "ca9c10bab52044e2bd277765fc8d6c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9"
I20260806 09:03:37.562759 30020 raft_consensus.cc:3037] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.566859 29732 raft_consensus.cc:2454] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9c10bab52044e2bd277765fc8d6c50 in term 1.
I20260806 09:03:37.567368 30020 raft_consensus.cc:2454] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9c10bab52044e2bd277765fc8d6c50 in term 1.
W20260806 09:03:37.567636 30072 tablet.cc:2369] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:37.567718 29807 leader_election.cc:302] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, ca9c10bab52044e2bd277765fc8d6c50; no voters:
I20260806 09:03:37.568387 30099 raft_consensus.cc:2781] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:37.569242 30099 raft_consensus.cc:686] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 1 LEADER]: Becoming Leader. State: Replica: ca9c10bab52044e2bd277765fc8d6c50, State: Running, Role: LEADER
I20260806 09:03:37.570405 30099 consensus_queue.cc:237] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.576881 29568 catalog_manager.cc:5495] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130). New cstate: current_term: 1 leader_uuid: "ca9c10bab52044e2bd277765fc8d6c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } health_report { overall_health: UNKNOWN } } }
W20260806 09:03:37.638489 29784 tablet.cc:2369] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:37.640647 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.631628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.27.187.130:38543)
0806 09:03:37.631872 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.631877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.631897 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:03:37.631997 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.632000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.632141 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.632346 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:37.632958 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.633904 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.636235 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.637032 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.637200 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.638026 (+ 826us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:37.639152 (+ 1126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:37.639335 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.639690 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.640029 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:03:37.652737 30020 raft_consensus.cc:1261] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Refusing update from remote peer ca9c10bab52044e2bd277765fc8d6c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:37.652787 29732 raft_consensus.cc:1261] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Refusing update from remote peer ca9c10bab52044e2bd277765fc8d6c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:37.654351 30102 consensus_queue.cc:1035] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:37.656520 30102 consensus_queue.cc:1035] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:37.673450 30106 mvcc.cc:205] Tried to move back new op lower bound from 7315484744289951744 to 7315484743973908480. Current Snapshot: MvccSnapshot[applied={T|T < 7315484744289951744}]
I20260806 09:03:37.678817 30109 mvcc.cc:205] Tried to move back new op lower bound from 7315484744289951744 to 7315484743973908480. Current Snapshot: MvccSnapshot[applied={T|T < 7315484744289951744}]
I20260806 09:03:37.704391 29571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:37.704773 29571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:37.709813 29571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Sending DeleteTablet for 3 replicas of tablet d0e0134c198549d89231258c3367a739
I20260806 09:03:37.711683 29856 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e0134c198549d89231258c3367a739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:37 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:34892
I20260806 09:03:37.711740 29712 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e0134c198549d89231258c3367a739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:37 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:49462
I20260806 09:03:37.712404 30000 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e0134c198549d89231258c3367a739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:37 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:35446
I20260806 09:03:37.716763 29571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:37.717851 29571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:37.722158 30116 tablet_replica.cc:330] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: stopping tablet replica
I20260806 09:03:37.722947 30116 raft_consensus.cc:2229] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:37.723578 30116 raft_consensus.cc:2258] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:37.730014 30115 tablet_replica.cc:330] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: stopping tablet replica
I20260806 09:03:37.730778 30115 raft_consensus.cc:2229] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:37.731441 30115 raft_consensus.cc:2258] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:37.732769 30117 tablet_replica.cc:330] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: stopping tablet replica
I20260806 09:03:37.733695 30116 ts_tablet_manager.cc:1905] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:37.734169 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc (DEFAULT_TABLE table=test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.734926 30000 tablet_service.cc:1467] Processing CreateTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc (DEFAULT_TABLE table=test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.735106 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b5d5ecbbda48e3bfff20962e0debdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.735529 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b5d5ecbbda48e3bfff20962e0debdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.743317 30116 ts_tablet_manager.cc:1918] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:03:37.743667 30116 log.cc:1196] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/d0e0134c198549d89231258c3367a739
I20260806 09:03:37.745491 30116 ts_tablet_manager.cc:1939] T d0e0134c198549d89231258c3367a739 P b966a98ea843438eaad46670378924e9: Deleting consensus metadata
I20260806 09:03:37.745858 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc (DEFAULT_TABLE table=test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:37.746965 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b5d5ecbbda48e3bfff20962e0debdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.749214 29556 catalog_manager.cc:4841] TS b966a98ea843438eaad46670378924e9 (127.27.187.131:37981): tablet d0e0134c198549d89231258c3367a739 (table test-table [id=4097e099ec914088a34e7283db9025e4]) successfully deleted
I20260806 09:03:37.750062 30091 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Bootstrap starting.
I20260806 09:03:37.750795 30092 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Bootstrap starting.
I20260806 09:03:37.753705 30091 tablet_bootstrap.cc:654] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.753974 30092 tablet_bootstrap.cc:654] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.758733 30117 raft_consensus.cc:2229] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:37.760012 30117 raft_consensus.cc:2258] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:37.760864 30092 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: No bootstrap required, opened a new log
I20260806 09:03:37.760986 30091 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: No bootstrap required, opened a new log
I20260806 09:03:37.761116 30092 ts_tablet_manager.cc:1397] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260806 09:03:37.761216 30091 ts_tablet_manager.cc:1397] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 09:03:37.762465 30092 raft_consensus.cc:348] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.762800 30092 raft_consensus.cc:374] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.762761 30091 raft_consensus.cc:348] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.762967 30092 raft_consensus.cc:729] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01197aa2622446c68cf592a4048d621e, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.763051 30091 raft_consensus.cc:374] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.763170 30091 raft_consensus.cc:729] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b966a98ea843438eaad46670378924e9, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.763465 30092 consensus_queue.cc:260] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.763636 30091 consensus_queue.cc:260] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.764686 30091 ts_tablet_manager.cc:1428] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 09:03:37.765882 30092 ts_tablet_manager.cc:1428] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:37.767228 30093 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Bootstrap starting.
I20260806 09:03:37.770533 30115 ts_tablet_manager.cc:1905] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:37.771109 30093 tablet_bootstrap.cc:654] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.772606 30117 ts_tablet_manager.cc:1905] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:37.780020 30093 tablet_bootstrap.cc:492] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: No bootstrap required, opened a new log
I20260806 09:03:37.780375 30093 ts_tablet_manager.cc:1397] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260806 09:03:37.782102 30093 raft_consensus.cc:348] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.782618 30115 ts_tablet_manager.cc:1918] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:03:37.782953 30115 log.cc:1196] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/d0e0134c198549d89231258c3367a739
I20260806 09:03:37.784123 30115 ts_tablet_manager.cc:1939] T d0e0134c198549d89231258c3367a739 P 01197aa2622446c68cf592a4048d621e: Deleting consensus metadata
I20260806 09:03:37.784261 30093 raft_consensus.cc:374] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.784888 30093 raft_consensus.cc:729] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9c10bab52044e2bd277765fc8d6c50, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.786221 30117 ts_tablet_manager.cc:1918] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:03:37.786481 30117 log.cc:1196] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/d0e0134c198549d89231258c3367a739
I20260806 09:03:37.786940 29556 catalog_manager.cc:4841] TS 01197aa2622446c68cf592a4048d621e (127.27.187.129:38951): tablet d0e0134c198549d89231258c3367a739 (table test-table [id=4097e099ec914088a34e7283db9025e4]) successfully deleted
I20260806 09:03:37.787441 30117 ts_tablet_manager.cc:1939] T d0e0134c198549d89231258c3367a739 P ca9c10bab52044e2bd277765fc8d6c50: Deleting consensus metadata
I20260806 09:03:37.787202 30093 consensus_queue.cc:260] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.788501 30093 ts_tablet_manager.cc:1428] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20260806 09:03:37.790995 29555 catalog_manager.cc:4841] TS ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543): tablet d0e0134c198549d89231258c3367a739 (table test-table [id=4097e099ec914088a34e7283db9025e4]) successfully deleted
I20260806 09:03:37.851917 30097 raft_consensus.cc:482] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:37.852464 30097 raft_consensus.cc:504] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.854760 30097 leader_election.cc:288] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:37.868028 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.855172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:54871 (local address 127.27.187.131:37981)
0806 09:03:37.855379 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.855387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.857776 (+ 2389us) server_negotiation.cc:407] Connection header received
0806 09:03:37.857967 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.857972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.858083 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.858208 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.859692 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.861235 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.865154 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.866397 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.866707 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.866843 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.867427 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.867628 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:03:37.868785 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.855266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:45673 (local address 127.27.187.130:38543)
0806 09:03:37.855549 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.855563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.857360 (+ 1797us) server_negotiation.cc:407] Connection header received
0806 09:03:37.857639 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.857666 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.857846 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.857949 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:37.859085 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.860877 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.865085 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.867101 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.867295 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.867390 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.867943 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.868121 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 09:03:37.868834 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.854989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38543 (local address 127.27.187.129:45673)
0806 09:03:37.857193 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.857215 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.857464 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.858151 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.858157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.858188 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.858861 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.858876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.861027 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.861033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.864759 (+ 3726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.864786 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.867502 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.867509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.867723 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.867759 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.867878 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.868066 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1955,"mutex_wait_us":1,"thread_start_us":2018,"threads_started":1}
I20260806 09:03:37.869786 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.855570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:37981 (local address 127.27.187.129:54871)
0806 09:03:37.857631 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.857652 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.857814 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.858288 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.858293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.858351 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:37.859358 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.859398 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.861422 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.861428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.864877 (+ 3449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.864892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.866846 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.866855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.867091 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.867120 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.868587 (+ 1467us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.868842 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":940,"threads_started":1}
I20260806 09:03:37.870779 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b5d5ecbbda48e3bfff20962e0debdc" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9" is_pre_election: true
I20260806 09:03:37.870471 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b5d5ecbbda48e3bfff20962e0debdc" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c10bab52044e2bd277765fc8d6c50" is_pre_election: true
I20260806 09:03:37.871093 30020 raft_consensus.cc:2454] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 0.
I20260806 09:03:37.871410 29877 raft_consensus.cc:2454] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 0.
I20260806 09:03:37.871856 29662 leader_election.cc:302] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, b966a98ea843438eaad46670378924e9; no voters:
I20260806 09:03:37.872574 30097 raft_consensus.cc:2781] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:37.872822 30097 raft_consensus.cc:482] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:37.872959 30097 raft_consensus.cc:3037] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.876132 30097 raft_consensus.cc:504] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.877187 30097 leader_election.cc:288] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 election: Requested vote from peers ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:37.877888 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b5d5ecbbda48e3bfff20962e0debdc" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c10bab52044e2bd277765fc8d6c50"
I20260806 09:03:37.877916 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b5d5ecbbda48e3bfff20962e0debdc" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9"
I20260806 09:03:37.878150 29877 raft_consensus.cc:3037] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.878208 30020 raft_consensus.cc:3037] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:37.880987 30020 raft_consensus.cc:2454] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 1.
I20260806 09:03:37.881548 29662 leader_election.cc:302] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, b966a98ea843438eaad46670378924e9; no voters:
I20260806 09:03:37.881989 30097 raft_consensus.cc:2781] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:37.883502 30097 raft_consensus.cc:686] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 1 LEADER]: Becoming Leader. State: Replica: 01197aa2622446c68cf592a4048d621e, State: Running, Role: LEADER
I20260806 09:03:37.883581 29877 raft_consensus.cc:2454] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 1.
I20260806 09:03:37.884213 30097 consensus_queue.cc:237] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.889345 29571 catalog_manager.cc:5495] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e reported cstate change: term changed from 0 to 1, leader changed from <none> to 01197aa2622446c68cf592a4048d621e (127.27.187.129). New cstate: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } health_report { overall_health: UNKNOWN } } }
Aug 06 09:03:37 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:37.959534 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.941265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34471 (local address 127.0.0.1:57284)
0806 09:03:37.941877 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:37.941901 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:37.942068 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:37.942708 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:37.942714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:37.943598 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:37.943874 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.943888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.945250 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.945256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.946752 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:37.946769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.947688 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.947695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:37.947905 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.948740 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:37.948767 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:37.951848 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:37.955701 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.955724 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.955734 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.956482 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.956975 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:37.956981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:37.956985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:37.957207 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:37.957945 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:37.957956 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:37.958199 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 09:03:37.958443 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 09:03:37.958833 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
I20260806 09:03:37.959594 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:37.941375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.27.187.190:34471)
0806 09:03:37.942351 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:37.942356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:37.942374 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:03:37.942424 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:37.942427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:37.942493 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:37.942597 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:37.944066 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.945073 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.946913 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:37.947472 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:37.947585 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:37.952192 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:37.952205 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:37.952209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:37.952275 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:37.955529 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.956684 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.956689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.956691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.956840 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:37.957354 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:37.957358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:37.957361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:37.957742 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:37.957933 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:37.958562 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 09:03:37.959084 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":858,"thread_start_us":709,"threads_started":1}
I20260806 09:03:37.966809 29571 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:57284:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:03:37.968230 29571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:37.977108 29711 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8f0f7879a4004b6495d38f039753a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d81215411fd3436bbdc2b4d4a2efad80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:37.977375 30000 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8f0f7879a4004b6495d38f039753a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d81215411fd3436bbdc2b4d4a2efad80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:37.977797 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8f0f7879a4004b6495d38f039753a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.977598 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb8f0f7879a4004b6495d38f039753a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d81215411fd3436bbdc2b4d4a2efad80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:37.978010 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8f0f7879a4004b6495d38f039753a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.978220 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb8f0f7879a4004b6495d38f039753a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:37.988099 30092 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: Bootstrap starting.
I20260806 09:03:37.988619 30091 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: Bootstrap starting.
I20260806 09:03:37.991356 30092 tablet_bootstrap.cc:654] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.991374 30091 tablet_bootstrap.cc:654] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.993216 30093 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: Bootstrap starting.
I20260806 09:03:37.995884 30093 tablet_bootstrap.cc:654] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:37.996135 30091 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: No bootstrap required, opened a new log
I20260806 09:03:37.996388 30091 ts_tablet_manager.cc:1397] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
W20260806 09:03:37.996894 29929 tablet.cc:2369] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:37.997475 30091 raft_consensus.cc:348] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.997673 30092 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: No bootstrap required, opened a new log
I20260806 09:03:37.997952 30091 raft_consensus.cc:374] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.998245 30092 ts_tablet_manager.cc:1397] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 09:03:37.998252 30091 raft_consensus.cc:729] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b966a98ea843438eaad46670378924e9, State: Initialized, Role: FOLLOWER
I20260806 09:03:37.998776 30091 consensus_queue.cc:260] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.999179 30092 raft_consensus.cc:348] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:37.999436 30092 raft_consensus.cc:374] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:37.999527 30092 raft_consensus.cc:729] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01197aa2622446c68cf592a4048d621e, State: Initialized, Role: FOLLOWER
I20260806 09:03:38.000489 30098 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.001056 30098 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.001171 30092 consensus_queue.cc:260] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.001740 30097 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.002396 30092 ts_tablet_manager.cc:1428] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 09:03:38.002547 30097 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.003358 30091 ts_tablet_manager.cc:1428] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 09:03:38.007817 30093 tablet_bootstrap.cc:492] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: No bootstrap required, opened a new log
I20260806 09:03:38.008095 30093 ts_tablet_manager.cc:1397] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260806 09:03:38.009156 30093 raft_consensus.cc:348] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.009470 30093 raft_consensus.cc:374] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:38.009563 30093 raft_consensus.cc:729] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9c10bab52044e2bd277765fc8d6c50, State: Initialized, Role: FOLLOWER
I20260806 09:03:38.010630 30093 consensus_queue.cc:260] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.011843 30099 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.012292 30099 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.013533 30093 ts_tablet_manager.cc:1428] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260806 09:03:38.095949 30097 raft_consensus.cc:482] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:38.096282 30097 raft_consensus.cc:504] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.097064 30097 leader_election.cc:288] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:38.097775 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8f0f7879a4004b6495d38f039753a" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9" is_pre_election: true
I20260806 09:03:38.097694 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8f0f7879a4004b6495d38f039753a" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c10bab52044e2bd277765fc8d6c50" is_pre_election: true
I20260806 09:03:38.098097 30020 raft_consensus.cc:2454] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 0.
I20260806 09:03:38.098140 29877 raft_consensus.cc:2454] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 0.
I20260806 09:03:38.098604 29662 leader_election.cc:302] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, b966a98ea843438eaad46670378924e9; no voters:
I20260806 09:03:38.098976 30097 raft_consensus.cc:2781] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:38.099118 30097 raft_consensus.cc:482] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:38.099262 30097 raft_consensus.cc:3037] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:38.101601 30097 raft_consensus.cc:504] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.102288 30097 leader_election.cc:288] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 election: Requested vote from peers ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543), b966a98ea843438eaad46670378924e9 (127.27.187.131:37981)
I20260806 09:03:38.102895 30020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8f0f7879a4004b6495d38f039753a" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b966a98ea843438eaad46670378924e9"
I20260806 09:03:38.103034 29877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb8f0f7879a4004b6495d38f039753a" candidate_uuid: "01197aa2622446c68cf592a4048d621e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c10bab52044e2bd277765fc8d6c50"
I20260806 09:03:38.103168 30020 raft_consensus.cc:3037] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:38.103300 29877 raft_consensus.cc:3037] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:38.107290 30020 raft_consensus.cc:2454] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 1.
I20260806 09:03:38.107307 29877 raft_consensus.cc:2454] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01197aa2622446c68cf592a4048d621e in term 1.
I20260806 09:03:38.107841 29662 leader_election.cc:302] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01197aa2622446c68cf592a4048d621e, b966a98ea843438eaad46670378924e9; no voters:
I20260806 09:03:38.108246 30097 raft_consensus.cc:2781] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:38.108546 30097 raft_consensus.cc:686] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [term 1 LEADER]: Becoming Leader. State: Replica: 01197aa2622446c68cf592a4048d621e, State: Running, Role: LEADER
I20260806 09:03:38.108866 30097 consensus_queue.cc:237] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } }
I20260806 09:03:38.110977 30124 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: TxnStatusTablet state changed. Reason: New leader 01197aa2622446c68cf592a4048d621e. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.111356 30124 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:38.112919 30137 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:03:38.113250 30137 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:03:38.113042 29571 catalog_manager.cc:5495] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e reported cstate change: term changed from 0 to 1, leader changed from <none> to 01197aa2622446c68cf592a4048d621e (127.27.187.129). New cstate: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } health_report { overall_health: UNKNOWN } } }
I20260806 09:03:38.143786 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.132061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38951 (local address 127.0.0.1:49478)
0806 09:03:38.132330 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:38.132358 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:38.132527 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:38.134346 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:38.134353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:38.134369 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:38.134755 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:38.134769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.137622 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.137630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:38.139780 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:38.139805 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.141172 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.141180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:38.141461 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.141557 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:38.143027 (+ 1470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:38.143058 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:03:38.143121 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 09:03:38.143314 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 09:03:38.144837 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.132494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.27.187.129:38951)
0806 09:03:38.133508 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.133514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.133574 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:03:38.133679 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.133685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.133807 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.134917 (+ 1110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:38.135546 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.137427 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.140107 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.140955 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.141769 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.141959 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:38.142883 (+ 924us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:38.144151 (+ 1268us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.144219 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.144322 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":6254,"threads_started":1}
I20260806 09:03:38.150612 30020 raft_consensus.cc:1261] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Refusing update from remote peer 01197aa2622446c68cf592a4048d621e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:38.150915 29877 raft_consensus.cc:1261] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Refusing update from remote peer 01197aa2622446c68cf592a4048d621e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:38.152001 30097 consensus_queue.cc:1035] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:38.152464 30124 consensus_queue.cc:1035] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:38.165644 30099 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: TxnStatusTablet state changed. Reason: New leader 01197aa2622446c68cf592a4048d621e. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.166281 30099 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.181023 30124 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.181423 30124 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:38.181985 30107 mvcc.cc:205] Tried to move back new op lower bound from 7315484746336247808 to 7315484746176696320. Current Snapshot: MvccSnapshot[applied={T|T < 7315484746336247808 or (T in {7315484746336247808})}]
I20260806 09:03:38.184700 30099 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.185038 30099 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P ca9c10bab52044e2bd277765fc8d6c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.188220 30098 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: TxnStatusTablet state changed. Reason: New leader 01197aa2622446c68cf592a4048d621e. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.188598 30098 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.192219 30097 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.192646 30097 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P 01197aa2622446c68cf592a4048d621e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:38.204322 30098 tablet_replica.cc:439] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01197aa2622446c68cf592a4048d621e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01197aa2622446c68cf592a4048d621e" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38951 } } peers { permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 } } peers { permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 } } }
I20260806 09:03:38.204681 30098 tablet_replica.cc:442] T 6fb8f0f7879a4004b6495d38f039753a P b966a98ea843438eaad46670378924e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:38.217547 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.208722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.27.187.129:38951)
0806 09:03:38.208906 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.208913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.209757 (+ 844us) server_negotiation.cc:407] Connection header received
0806 09:03:38.210098 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.210107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.210232 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.210313 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:38.211214 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.212436 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.214879 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.215606 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.215734 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.216493 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:38.216675 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:38.216749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.217003 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.217104 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:03:38.243070 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.233542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:52189 (local address 127.27.187.129:38951)
0806 09:03:38.233816 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.233824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.235037 (+ 1213us) server_negotiation.cc:407] Connection header received
0806 09:03:38.235277 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.235284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.235445 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.235560 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:38.236695 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.238048 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.240530 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.241248 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.241420 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.242092 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:38.242236 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:38.242330 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.242523 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.242622 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:03:38.243610 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.233797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38951 (local address 127.27.187.129:52189)
0806 09:03:38.234886 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:38.234916 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:38.235110 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:38.235765 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:38.235770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:38.235784 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:38.236448 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:38.236470 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.238262 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.238268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:38.240336 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:38.240358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.241524 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.241534 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:38.241773 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.241877 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:38.242365 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:38.242379 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 09:03:38.242495 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:03:38.242670 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":530,"threads_started":1}
I20260806 09:03:38.258890 29877 raft_consensus.cc:1261] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Refusing update from remote peer 01197aa2622446c68cf592a4048d621e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:38.258965 30019 raft_consensus.cc:1261] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Refusing update from remote peer 01197aa2622446c68cf592a4048d621e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:38.260069 30141 consensus_queue.cc:1035] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9c10bab52044e2bd277765fc8d6c50" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:38.260612 30124 consensus_queue.cc:1035] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b966a98ea843438eaad46670378924e9" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:38.410777 29570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:38.411253 29570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:38.416617 29570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf75b40e57ec4e698392113a25bba3d5: Sending DeleteTablet for 3 replicas of tablet 50b5d5ecbbda48e3bfff20962e0debdc
I20260806 09:03:38.417779 29711 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:38 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:49462
I20260806 09:03:38.419075 29855 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:38 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:34892
I20260806 09:03:38.427462 30000 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b5d5ecbbda48e3bfff20962e0debdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:38 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:35446
I20260806 09:03:38.427649 30155 tablet_replica.cc:330] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: stopping tablet replica
I20260806 09:03:38.428143 30155 raft_consensus.cc:2229] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:38.428719 30155 raft_consensus.cc:2258] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:38.433635 30155 ts_tablet_manager.cc:1905] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:38.443804 30156 tablet_replica.cc:330] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: stopping tablet replica
I20260806 09:03:38.444069 30156 raft_consensus.cc:2229] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:38.444478 30156 raft_consensus.cc:2258] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:38.445643 30154 tablet_replica.cc:330] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: stopping tablet replica
I20260806 09:03:38.445945 30154 raft_consensus.cc:2229] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:38.447506 30156 ts_tablet_manager.cc:1905] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:38.450830 30154 raft_consensus.cc:2258] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:38.453838 30154 ts_tablet_manager.cc:1905] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:38.455715 30156 ts_tablet_manager.cc:1918] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:38.456048 30156 log.cc:1196] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/50b5d5ecbbda48e3bfff20962e0debdc
I20260806 09:03:38.456586 30156 ts_tablet_manager.cc:1939] T 50b5d5ecbbda48e3bfff20962e0debdc P b966a98ea843438eaad46670378924e9: Deleting consensus metadata
I20260806 09:03:38.456980 30155 ts_tablet_manager.cc:1918] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:38.457173 30155 log.cc:1196] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/50b5d5ecbbda48e3bfff20962e0debdc
I20260806 09:03:38.457829 30155 ts_tablet_manager.cc:1939] T 50b5d5ecbbda48e3bfff20962e0debdc P ca9c10bab52044e2bd277765fc8d6c50: Deleting consensus metadata
I20260806 09:03:38.458115 29556 catalog_manager.cc:4841] TS b966a98ea843438eaad46670378924e9 (127.27.187.131:37981): tablet 50b5d5ecbbda48e3bfff20962e0debdc (table test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]) successfully deleted
I20260806 09:03:38.460698 29555 catalog_manager.cc:4841] TS ca9c10bab52044e2bd277765fc8d6c50 (127.27.187.130:38543): tablet 50b5d5ecbbda48e3bfff20962e0debdc (table test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]) successfully deleted
I20260806 09:03:38.466243 30154 ts_tablet_manager.cc:1918] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:38.466568 30154 log.cc:1196] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/50b5d5ecbbda48e3bfff20962e0debdc
I20260806 09:03:38.467345 30154 ts_tablet_manager.cc:1939] T 50b5d5ecbbda48e3bfff20962e0debdc P 01197aa2622446c68cf592a4048d621e: Deleting consensus metadata
I20260806 09:03:38.468979 29556 catalog_manager.cc:4841] TS 01197aa2622446c68cf592a4048d621e (127.27.187.129:38951): tablet 50b5d5ecbbda48e3bfff20962e0debdc (table test-table [id=b12e07e9725e4d15bb0bfe71474dca7f]) successfully deleted
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:38.482072 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.444538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.27.187.129:38951)
0806 09:03:38.444809 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.444817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.459513 (+ 14696us) server_negotiation.cc:407] Connection header received
0806 09:03:38.459780 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.459786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.459878 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.459982 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:38.462840 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.464157 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.466424 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.467236 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.467356 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.473595 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:38.473622 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:38.473647 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:38.473702 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:38.477996 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.479197 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.479209 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.479213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.479415 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.479921 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.479927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.479930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.480465 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:38.480857 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.481217 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.481464 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:03:38.483129 29778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49496
I20260806 09:03:38.509929 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.486758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.27.187.190:34471)
0806 09:03:38.487683 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.487688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.487705 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:03:38.487761 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.487764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.487827 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.487922 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:38.491433 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.493237 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.496348 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.497012 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.497197 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.502335 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:38.502341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:38.502345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:38.502430 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:38.506360 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.507448 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.507453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.507455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.507684 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.508143 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.508146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.508148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.508494 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:38.508749 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.509229 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.509469 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":646,"threads_started":1}
W20260806 09:03:38.510666 29570 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57300
I20260806 09:03:38.513007 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29644
I20260806 09:03:38.530815 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.514447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.27.187.129:38951)
0806 09:03:38.514609 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.514614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.515266 (+ 652us) server_negotiation.cc:407] Connection header received
0806 09:03:38.515521 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.515527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.515608 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.515728 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:38.517096 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.517881 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.519570 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.520124 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.520279 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.523490 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:38.523495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:38.523499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:38.523535 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:38.527065 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.527904 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.527910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.527914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.528170 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.528681 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.528686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.528690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.529089 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:38.529538 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.529839 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.530171 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 09:03:38.531499 29778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49506
W20260806 09:03:38.532356 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29644: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:38.532698 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29644
I20260806 09:03:38.555670 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29787
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:03:38.573028 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.556589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.27.187.130:38543)
0806 09:03:38.557223 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.557227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.557245 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:03:38.557303 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.557307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.557367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.557466 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:38.558825 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.559537 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.560866 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.561300 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.561440 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.566436 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:38.566453 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:38.566470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:38.566515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:38.569736 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.570618 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.570626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.570628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.570874 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.571285 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.571290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.571295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.571709 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:38.572041 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.572376 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.572646 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":351,"threads_started":1}
W20260806 09:03:38.573822 29923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34920
W20260806 09:03:38.574657 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:38.574924 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29787
I20260806 09:03:38.599385 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29932
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:03:38.617262 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:38.600262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.27.187.131:37981)
0806 09:03:38.600782 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:38.600787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:38.600804 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:03:38.600862 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:38.600866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:38.600927 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:38.601028 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:38.602422 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.603326 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.604701 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:38.605218 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:38.605336 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:38.610319 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:38.610344 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:38.610364 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:38.610409 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:38.613747 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.614971 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.614981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.614983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.615190 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:38.615610 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:38.615616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:38.615619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:38.616037 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:38.616308 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:38.616557 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 09:03:38.616757 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":286,"threads_started":1}
W20260806 09:03:38.618062 30066 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35456
W20260806 09:03:38.618973 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29932: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:38.619123 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29932
I20260806 09:03:38.638337 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 29538
I20260806 09:03:38.639631 29613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:38.763170 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 29538
2026-08-06T09:03:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5236 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[30168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[30168](info): set up 2 sockets
Aug 06 09:03:38 dist-test-slave-8tdl krb5kdc[30168](info): commencing operation
krb5kdc: starting...
W20260806 09:03:39.929765 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:40 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:40.171566 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44691
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:40.528162 30181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:40.528563 30181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:40.528672 30181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:40.536993 30181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:40.537135 30181 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:03:40.537180 30181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:40.537237 30181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:40.537282 30181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:40.537352 30181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:40.548084 30181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:44691
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:40.549451 30181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:40.551440 30181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:40.564308 30186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:40.564275 30187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:40.564396 30189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:40.565551 30181 server_base.cc:1034] running on GCE node
I20260806 09:03:40.566406 30181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:40.568043 30181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:40.569401 30181 hybrid_clock.cc:648] HybridClock initialized: now 1786007020569333 us; error 37 us; skew 500 ppm
I20260806 09:03:40.569808 30181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:40 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007020, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:40.579708 30181 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:40.581820 30181 webserver.cc:460] Webserver started at http://127.27.187.190:40925/ using document root <none> and password file <none>
I20260806 09:03:40.582667 30181 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:40.582860 30181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:40.583159 30181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:40.586387 30181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "f0719b323a5c426fbcddff91ad72038b"
format_stamp: "Formatted at 2026-08-06 09:03:40 on dist-test-slave-8tdl"
server_key: "96df1865c1ae1e017f3893790f6f5434"
server_key_iv: "ef1ddb1ec6f02b6420a014c347fce667"
server_key_version: "encryptionkey@0"
I20260806 09:03:40.587278 30181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "f0719b323a5c426fbcddff91ad72038b"
format_stamp: "Formatted at 2026-08-06 09:03:40 on dist-test-slave-8tdl"
server_key: "96df1865c1ae1e017f3893790f6f5434"
server_key_iv: "ef1ddb1ec6f02b6420a014c347fce667"
server_key_version: "encryptionkey@0"
I20260806 09:03:40.593092 30181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 09:03:40.597050 30196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:40.598773 30181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:03:40.598937 30181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "f0719b323a5c426fbcddff91ad72038b"
format_stamp: "Formatted at 2026-08-06 09:03:40 on dist-test-slave-8tdl"
server_key: "96df1865c1ae1e017f3893790f6f5434"
server_key_iv: "ef1ddb1ec6f02b6420a014c347fce667"
server_key_version: "encryptionkey@0"
I20260806 09:03:40.599143 30181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:40.615278 30181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:40.619943 30181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:40.620208 30181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:40.644318 30181 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:44691
I20260806 09:03:40.644337 30258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:44691 every 8 connection(s)
I20260806 09:03:40.645895 30181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:40.650214 30259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:40.656293 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 30181
I20260806 09:03:40.656531 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:40.657439 28398 external_mini_cluster.cc:1428] Setting key bcf5324feb84342b5512b95325457e1e
I20260806 09:03:40.663085 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Bootstrap starting.
I20260806 09:03:40.667219 30259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:40.668640 30259 log.cc:824] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:40.673270 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: No bootstrap required, opened a new log
Aug 06 09:03:40 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:40.681843 30259 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } }
I20260806 09:03:40.682333 30259 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:40.682462 30259 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0719b323a5c426fbcddff91ad72038b, State: Initialized, Role: FOLLOWER
I20260806 09:03:40.683261 30259 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } }
I20260806 09:03:40.683566 30259 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:40.683735 30259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:40.683943 30259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:40.684762 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:40.659896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.27.187.190:44691)
0806 09:03:40.660808 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:40.660821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:40.660873 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:03:40.662751 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:40.662777 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:40.663252 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:40.663631 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:40.665605 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:40.667045 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:40.668933 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:40.670188 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:40.670362 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:40.675859 (+ 5497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:40.675871 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:40.675889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:40.675944 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:40.679668 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:40.680493 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:40.680500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:40.680507 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:40.680699 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:40.681173 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:40.681179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:40.681183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:40.681796 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:40.682129 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:40.682649 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 09:03:40.683176 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":447,"threads_started":1}
I20260806 09:03:40.687352 30259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } }
I20260806 09:03:40.687958 30259 leader_election.cc:302] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0719b323a5c426fbcddff91ad72038b; no voters:
I20260806 09:03:40.688691 30259 leader_election.cc:288] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:40.688889 30264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:40.689692 30264 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [term 1 LEADER]: Becoming Leader. State: Replica: f0719b323a5c426fbcddff91ad72038b, State: Running, Role: LEADER
I20260806 09:03:40.690387 30264 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } }
I20260806 09:03:40.691042 30259 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:40.693529 30266 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0719b323a5c426fbcddff91ad72038b. Latest consensus state: current_term: 1 leader_uuid: "f0719b323a5c426fbcddff91ad72038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } } }
I20260806 09:03:40.694278 30266 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:40.693495 30265 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0719b323a5c426fbcddff91ad72038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0719b323a5c426fbcddff91ad72038b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44691 } } }
I20260806 09:03:40.694828 30265 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:40.696307 30271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:40.704561 30271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:40.710423 30282 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:03:40.716809 30271 catalog_manager.cc:1347] Generated new cluster ID: 7ea431ba38bd4434b7fd6c1eacf0a50b
I20260806 09:03:40.716944 30271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:40.733554 30271 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:40.735638 30271 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:40.749182 30271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Generated new TSK 0
I20260806 09:03:40.750211 30271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:40.824061 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44691
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:41.200971 30287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:41.201344 30287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:41.201570 30287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:41.209510 30287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:41.209762 30287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:41.210001 30287 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:03:41.210108 30287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:41.210271 30287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:41.221177 30287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:41.222721 30287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:41.224174 30287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:41.238555 30292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:41.238494 30293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:41.238456 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:41.239593 30287 server_base.cc:1034] running on GCE node
I20260806 09:03:41.240355 30287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:41.241453 30287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:41.242770 30287 hybrid_clock.cc:648] HybridClock initialized: now 1786007021242706 us; error 100 us; skew 500 ppm
I20260806 09:03:41.243139 30287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:41 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007021, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:41.246871 30287 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:41.248804 30287 webserver.cc:460] Webserver started at http://127.27.187.129:45661/ using document root <none> and password file <none>
I20260806 09:03:41.249816 30287 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:41.249985 30287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:41.250363 30287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:41.253973 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "05bd647cd6054e6982f04d8d85246a0a"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "0914be03123f0ac191c0078beefa0aed"
server_key_iv: "ab77173afcdc66942ce14628617c3fdd"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.254966 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "05bd647cd6054e6982f04d8d85246a0a"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "0914be03123f0ac191c0078beefa0aed"
server_key_iv: "ab77173afcdc66942ce14628617c3fdd"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.261300 30287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:03:41.265417 30302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:41.267300 30287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:03:41.267515 30287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "05bd647cd6054e6982f04d8d85246a0a"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "0914be03123f0ac191c0078beefa0aed"
server_key_iv: "ab77173afcdc66942ce14628617c3fdd"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.267745 30287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:41.307591 30287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:41.312938 30287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:41.313328 30287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:41.316428 30287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:41.316576 30287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:41.316708 30287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:41.316818 30287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:41 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007021, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:41.347167 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.317102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:53397 (local address 127.27.187.190:44691)
0806 09:03:41.317825 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:41.317833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:41.319136 (+ 1303us) server_negotiation.cc:407] Connection header received
0806 09:03:41.321845 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:41.321852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:41.321955 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:41.322091 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:41.325000 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.326290 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.329664 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.330494 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.330670 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.337259 (+ 6589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:41.337266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:41.337269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:41.337313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:41.341587 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.343107 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.343114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.343118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.343368 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.344164 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.344171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.344174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.344709 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:41.344901 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:41.346303 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0806 09:03:41.346642 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":456,"threads_started":1}
I20260806 09:03:41.349190 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.317935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.129:53397)
0806 09:03:41.318931 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:41.318984 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:41.321388 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:41.322501 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:41.322517 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:41.323493 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:41.324659 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.324696 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.326551 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.326557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.329397 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.329425 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.330893 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.330906 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.331291 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.332187 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:41.332252 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:41.336815 (+ 4563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:41.341843 (+ 5028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.341861 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.341891 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.342926 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.343546 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.343555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.343562 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.343992 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.344897 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:41.344911 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:41.345676 (+ 765us) client_negotiation.cc:769] Sending connection context
0806 09:03:41.346125 (+ 449us) client_negotiation.cc:240] Negotiation successful
0806 09:03:41.346992 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":437,"threads_started":1}
I20260806 09:03:41.360579 30287 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:46045
I20260806 09:03:41.360682 30424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:46045 every 8 connection(s)
I20260806 09:03:41.362054 30287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:41.363237 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 30287
I20260806 09:03:41.363480 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:41.364212 28398 external_mini_cluster.cc:1428] Setting key 233e9429381520ebbbea2da1c4d020c7
I20260806 09:03:41.364619 30425 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44691
I20260806 09:03:41.364921 30425 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:41.365855 30425 heartbeater.cc:511] Master 127.27.187.190:44691 requested a full tablet report, sending...
I20260806 09:03:41.368611 30213 ts_manager.cc:194] Registered new tserver with Master: 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045)
I20260806 09:03:41.370870 30213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:53397
I20260806 09:03:41.376448 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.357821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34075 (local address 127.27.187.190:44691)
0806 09:03:41.358061 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:41.358066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:41.358769 (+ 703us) server_negotiation.cc:407] Connection header received
0806 09:03:41.359038 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:41.359044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:41.359124 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:41.359238 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:41.360698 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.361514 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.365905 (+ 4391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.366612 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.366771 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.369282 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:41.369289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:41.369293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:41.369330 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:41.372870 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.373840 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.373846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.373849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.374017 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.374493 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.374501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.374506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.375005 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:41.375173 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:41.375665 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 09:03:41.375968 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:03:41.376475 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.357862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.129:34075)
0806 09:03:41.358611 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:41.358628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:41.358869 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:41.359273 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:41.359278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:41.360005 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:41.360432 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.360451 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.361709 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.361713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.365684 (+ 3971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.365705 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.367016 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.367020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.367191 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.367902 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:41.367926 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:41.369031 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:41.373051 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.373056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.373061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.373682 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.374178 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.374183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.374186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.374349 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.375166 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:41.375171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:41.375354 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 09:03:41.375556 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 09:03:41.375968 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":465,"threads_started":1}
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:41.420166 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44691
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:41.786602 30429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:41.787014 30429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:41.787125 30429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:41.794999 30429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:41.795125 30429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:41.795233 30429 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:03:41.795282 30429 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:41.795356 30429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:41.806172 30429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:41.807616 30429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:41.809010 30429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:41.823798 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:41.823781 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:41.823765 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:41.824747 30429 server_base.cc:1034] running on GCE node
I20260806 09:03:41.825551 30429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:41.827097 30429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:41.828380 30429 hybrid_clock.cc:648] HybridClock initialized: now 1786007021828336 us; error 52 us; skew 500 ppm
I20260806 09:03:41.828764 30429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:41 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007021, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:41.832840 30429 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:41.835035 30429 webserver.cc:460] Webserver started at http://127.27.187.130:40827/ using document root <none> and password file <none>
I20260806 09:03:41.835924 30429 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:41.836094 30429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:41.836470 30429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:41.839908 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "3aa7b13c013f4460b3d59a4616dd17df"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "301ebeb35a6f245aeee14bc5635b4a53"
server_key_iv: "6e66e3cd2b9635f43bf9240fa4eeaa59"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.840852 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "3aa7b13c013f4460b3d59a4616dd17df"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "301ebeb35a6f245aeee14bc5635b4a53"
server_key_iv: "6e66e3cd2b9635f43bf9240fa4eeaa59"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.847254 30429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:03:41.851668 30444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:41.853492 30429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:03:41.853745 30429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "3aa7b13c013f4460b3d59a4616dd17df"
format_stamp: "Formatted at 2026-08-06 09:03:41 on dist-test-slave-8tdl"
server_key: "301ebeb35a6f245aeee14bc5635b4a53"
server_key_iv: "6e66e3cd2b9635f43bf9240fa4eeaa59"
server_key_version: "encryptionkey@0"
I20260806 09:03:41.853956 30429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:41.871207 30429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:41.876403 30429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:41.876683 30429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:41.880057 30429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:41.880246 30429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:41.880389 30429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:41.880487 30429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:41 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007021, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:41.911175 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.880874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:33247 (local address 127.27.187.190:44691)
0806 09:03:41.881765 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:41.881773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:41.882952 (+ 1179us) server_negotiation.cc:407] Connection header received
0806 09:03:41.885696 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:41.885705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:41.885835 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:41.885968 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:41.888702 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.889859 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.893671 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.894223 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.894403 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.900479 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:41.900489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:41.900498 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:41.900563 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:41.905847 (+ 5284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.907417 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.907422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.907426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.907634 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.908493 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.908500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.908504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.908907 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:41.909126 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:41.910296 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0806 09:03:41.910603 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":618,"threads_started":1}
I20260806 09:03:41.913417 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.881641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.130:33247)
0806 09:03:41.882734 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:41.882787 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:41.885131 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:41.886287 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:41.886305 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:41.887362 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:41.888439 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.888479 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.890123 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.890130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.893417 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.893440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.894577 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.894592 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.895059 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.896094 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:41.896138 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:41.900064 (+ 3926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:41.906107 (+ 6043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.906123 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.906150 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.907162 (+ 1012us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.907810 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.907817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.907821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.908219 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.909087 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:41.909097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:41.909691 (+ 594us) client_negotiation.cc:769] Sending connection context
0806 09:03:41.910163 (+ 472us) client_negotiation.cc:240] Negotiation successful
0806 09:03:41.910981 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":437,"threads_started":1}
I20260806 09:03:41.930512 30429 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:44083
I20260806 09:03:41.930567 30566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:44083 every 8 connection(s)
I20260806 09:03:41.932040 30429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:41.934928 30567 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44691
I20260806 09:03:41.935398 30567 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:41.935915 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 30429
I20260806 09:03:41.936107 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:41.936400 30567 heartbeater.cc:511] Master 127.27.187.190:44691 requested a full tablet report, sending...
I20260806 09:03:41.936743 28398 external_mini_cluster.cc:1428] Setting key 1a34949970450e70c4cb61ef49716079
I20260806 09:03:41.938104 30213 ts_manager.cc:194] Registered new tserver with Master: 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:41.938551 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.920651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.130:59113)
0806 09:03:41.921572 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:41.921596 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:41.921808 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:41.922226 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:41.922231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:41.923076 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:41.923724 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.923745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.925630 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.925635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.927734 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:41.927752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.928697 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.928704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:41.928891 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.929705 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:41.929727 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:41.930853 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:41.935100 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.935107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.935111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.935852 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.936370 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:41.936375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:41.936380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:41.936550 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:41.937327 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:41.937332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:41.937571 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 09:03:41.937732 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 09:03:41.938087 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":536,"threads_started":1}
I20260806 09:03:41.939224 30213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:33247
I20260806 09:03:41.939587 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:41.920603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:59113 (local address 127.27.187.190:44691)
0806 09:03:41.920838 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:41.920848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:41.921712 (+ 864us) server_negotiation.cc:407] Connection header received
0806 09:03:41.921964 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:41.921971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:41.922064 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:41.922186 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:41.924020 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.925413 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.927940 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:41.928457 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:41.928626 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:41.931099 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:41.931110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:41.931115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:41.931166 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:41.934949 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.936046 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.936052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.936055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.936228 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:41.936709 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:41.936717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:41.936723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:41.937209 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:41.937373 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:41.938923 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0806 09:03:41.939167 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:41.992738 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44691
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:42.370016 30571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:42.370337 30571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:42.370520 30571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:03:42.375388 30425 heartbeater.cc:503] Master 127.27.187.190:44691 was elected leader, sending a full tablet report...
W20260806 09:03:42.378926 30571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:42.379046 30571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:42.379189 30571 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:03:42.379312 30571 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:03:42.379429 30571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:42.390079 30571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:42.391619 30571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:42.393131 30571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:42.407873 30576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:42.408025 30579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:42.408041 30577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:42.409471 30571 server_base.cc:1034] running on GCE node
I20260806 09:03:42.410310 30571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:42.411487 30571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:42.412905 30571 hybrid_clock.cc:648] HybridClock initialized: now 1786007022412813 us; error 64 us; skew 500 ppm
I20260806 09:03:42.413367 30571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:42 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007022, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:42.417378 30571 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:42.419260 30571 webserver.cc:460] Webserver started at http://127.27.187.131:41613/ using document root <none> and password file <none>
I20260806 09:03:42.420010 30571 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:42.420120 30571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:42.420408 30571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:42.423599 30571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "9a80ec3302c84886aec940727d839808"
format_stamp: "Formatted at 2026-08-06 09:03:42 on dist-test-slave-8tdl"
server_key: "3610b7bad72b116480b8406df0aee17d"
server_key_iv: "796ab099716c83af1777bf2c12ccf254"
server_key_version: "encryptionkey@0"
I20260806 09:03:42.424378 30571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "9a80ec3302c84886aec940727d839808"
format_stamp: "Formatted at 2026-08-06 09:03:42 on dist-test-slave-8tdl"
server_key: "3610b7bad72b116480b8406df0aee17d"
server_key_iv: "796ab099716c83af1777bf2c12ccf254"
server_key_version: "encryptionkey@0"
I20260806 09:03:42.429968 30571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:03:42.434247 30586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:42.436096 30571 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:42.436311 30571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "9a80ec3302c84886aec940727d839808"
format_stamp: "Formatted at 2026-08-06 09:03:42 on dist-test-slave-8tdl"
server_key: "3610b7bad72b116480b8406df0aee17d"
server_key_iv: "796ab099716c83af1777bf2c12ccf254"
server_key_version: "encryptionkey@0"
I20260806 09:03:42.436522 30571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:42.452765 30571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:42.457736 30571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:42.458129 30571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:42.461063 30571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:42.461223 30571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:42.461371 30571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:42.461480 30571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:03:42 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007022, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:42.489351 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.462318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:55507 (local address 127.27.187.190:44691)
0806 09:03:42.462946 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.462952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.464139 (+ 1187us) server_negotiation.cc:407] Connection header received
0806 09:03:42.466426 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.466432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.466560 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.466711 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:42.470135 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.471304 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.474824 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.475456 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.475631 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.481147 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:42.481158 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:42.481166 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:42.481223 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:42.485338 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.486457 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.486465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.486469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.486719 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.487343 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.487348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.487352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.487671 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:42.487831 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.488672 (+ 841us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.488927 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":383,"threads_started":1}
I20260806 09:03:42.490459 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.463004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.131:55507)
0806 09:03:42.463948 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.463994 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.466097 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.467214 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.467227 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.468502 (+ 1275us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:42.469835 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.469877 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.471524 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.471531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.474578 (+ 3047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.474610 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.475823 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.475833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.476087 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.477109 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:42.477152 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:42.480763 (+ 3611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:42.485523 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.485542 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.485574 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.486243 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.486882 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.486889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.486893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.487198 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.487821 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:42.487830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:42.488273 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.488555 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.489152 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":452,"threads_started":1}
I20260806 09:03:42.513813 30571 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:42503
I20260806 09:03:42.513934 30711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:42503 every 8 connection(s)
I20260806 09:03:42.515377 30571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:42.519645 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 30571
I20260806 09:03:42.519871 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:42.520427 28398 external_mini_cluster.cc:1428] Setting key 1c3a9d90fd013b4eaa926a47da84cb57
I20260806 09:03:42.522704 30712 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44691
I20260806 09:03:42.522989 30712 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:42.523945 30712 heartbeater.cc:511] Master 127.27.187.190:44691 requested a full tablet report, sending...
I20260806 09:03:42.525614 30706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.510210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.27.187.131:58921)
0806 09:03:42.511144 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.511167 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.511369 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.511816 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.511820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.512398 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:42.512938 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.512961 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.514497 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.514503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.515746 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.515765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.516592 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.516598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.516835 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.517538 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:42.517556 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:42.518330 (+ 774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:42.522593 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.522600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.522604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.523286 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.523784 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.523791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.523796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.523977 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.524557 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:42.524564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:42.524771 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.524900 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.525300 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":383,"threads_started":1}
I20260806 09:03:42.525583 30213 ts_manager.cc:194] Registered new tserver with Master: 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503)
I20260806 09:03:42.525689 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.510161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:58921 (local address 127.27.187.190:44691)
0806 09:03:42.510424 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.510430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.511277 (+ 847us) server_negotiation.cc:407] Connection header received
0806 09:03:42.511530 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.511536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.511637 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.511779 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:42.513126 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.514336 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.515908 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.516389 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.516538 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.518555 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:42.518560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:42.518563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:42.518596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:42.522419 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.523425 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.523430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.523434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.523626 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.524123 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.524128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.524131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.524425 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:42.524550 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.525058 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.525340 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260806 09:03:42.526633 30213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:55507
I20260806 09:03:42.537220 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:03:42 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:03:42 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:42.595821 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.579973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.27.187.190:44691)
0806 09:03:42.580163 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.580170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.580545 (+ 375us) server_negotiation.cc:407] Connection header received
0806 09:03:42.580729 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.580733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.580808 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.580921 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:42.582565 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.583519 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.584892 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.585316 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.585435 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.589641 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:42.589653 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:42.589656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:42.589695 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:42.592891 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.593748 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.593753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.593755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.593903 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.594460 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.594464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.594466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.594847 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:42.595080 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.595375 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.595552 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:03:42.600730 30213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:42.609133 30213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:42.649379 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.622991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46045 (local address 127.0.0.1:53494)
0806 09:03:42.637354 (+ 14363us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.637442 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.637733 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.639901 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.639909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.639938 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:42.640416 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.640435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.642480 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.642486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.645094 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.645108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.648176 (+ 3068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.648184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.648480 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.648553 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.648728 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.648965 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13927,"thread_start_us":15458,"threads_started":1}
I20260806 09:03:42.650826 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.623265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.27.187.129:46045)
0806 09:03:42.638496 (+ 15231us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.638506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.638540 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:03:42.638647 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.638655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.638974 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.639185 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:42.640631 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.642254 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.645285 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.647622 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.647824 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.648027 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.650205 (+ 2178us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.650416 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15055,"thread_start_us":14890,"threads_started":1}
I20260806 09:03:42.651273 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.623245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42503 (local address 127.0.0.1:42350)
0806 09:03:42.641554 (+ 18309us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.641578 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.641816 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.642922 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.642928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.642962 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:42.643453 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.643466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.645723 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.645729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.647917 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.647933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.650259 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.650266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.650533 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.650567 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.650649 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.650820 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18145,"mutex_wait_us":28,"thread_start_us":18510,"threads_started":1}
I20260806 09:03:42.652268 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.623802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.27.187.131:42503)
0806 09:03:42.639990 (+ 16188us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.640003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.641680 (+ 1677us) server_negotiation.cc:407] Connection header received
0806 09:03:42.642004 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.642013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.642291 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.642506 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:42.643735 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.645536 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.648117 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.649689 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.649867 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.650048 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.651645 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.651841 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16004,"thread_start_us":15822,"threads_started":1}
I20260806 09:03:42.655138 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.623611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:44083 (local address 127.0.0.1:55586)
0806 09:03:42.642034 (+ 18423us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.642051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.642231 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.643188 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.643193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.643220 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:42.645624 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.645638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.648256 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.648260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.650536 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.650555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.654242 (+ 3687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.654250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.654445 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.654480 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.654576 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.654744 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18306,"mutex_wait_us":1966,"thread_start_us":17627,"threads_started":1}
I20260806 09:03:42.655517 30353 tablet_service.cc:1467] Processing CreateTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 (DEFAULT_TABLE table=test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:42.656358 30639 tablet_service.cc:1467] Processing CreateTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 (DEFAULT_TABLE table=test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:42.657404 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78024efd9c3948e99a1ef1e02bdfba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.657670 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.624126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.27.187.130:44083)
0806 09:03:42.641028 (+ 16902us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.641037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.642191 (+ 1154us) server_negotiation.cc:407] Connection header received
0806 09:03:42.642423 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.642428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.642668 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.642942 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:42.645868 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.648065 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.650771 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.653705 (+ 2934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.653884 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.654053 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.655491 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.655675 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16744,"thread_start_us":16588,"threads_started":1}
I20260806 09:03:42.657819 30639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78024efd9c3948e99a1ef1e02bdfba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.661746 30495 tablet_service.cc:1467] Processing CreateTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 (DEFAULT_TABLE table=test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:42.665066 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78024efd9c3948e99a1ef1e02bdfba11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.673161 30731 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Bootstrap starting.
I20260806 09:03:42.676911 30731 tablet_bootstrap.cc:654] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.678292 30731 log.cc:824] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:42.679437 30732 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Bootstrap starting.
I20260806 09:03:42.682427 30734 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Bootstrap starting.
I20260806 09:03:42.683116 30731 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: No bootstrap required, opened a new log
I20260806 09:03:42.683461 30731 ts_tablet_manager.cc:1397] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 09:03:42.684088 30732 tablet_bootstrap.cc:654] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.685210 30732 log.cc:824] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:42.686204 30734 tablet_bootstrap.cc:654] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.687493 30734 log.cc:824] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:42.688205 30732 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: No bootstrap required, opened a new log
I20260806 09:03:42.688524 30732 ts_tablet_manager.cc:1397] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 09:03:42.690886 30734 tablet_bootstrap.cc:492] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: No bootstrap required, opened a new log
I20260806 09:03:42.691273 30734 ts_tablet_manager.cc:1397] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260806 09:03:42.694547 30731 raft_consensus.cc:348] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.695087 30731 raft_consensus.cc:374] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.695235 30731 raft_consensus.cc:729] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a80ec3302c84886aec940727d839808, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.695943 30731 consensus_queue.cc:260] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.697891 30731 ts_tablet_manager.cc:1428] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 09:03:42.697860 30732 raft_consensus.cc:348] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.698463 30712 heartbeater.cc:503] Master 127.27.187.190:44691 was elected leader, sending a full tablet report...
I20260806 09:03:42.698545 30732 raft_consensus.cc:374] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.698669 30732 raft_consensus.cc:729] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.699609 30732 consensus_queue.cc:260] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.702183 30732 ts_tablet_manager.cc:1428] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 09:03:42.702265 30734 raft_consensus.cc:348] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.702900 30734 raft_consensus.cc:374] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.702982 30738 raft_consensus.cc:482] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:42.703059 30734 raft_consensus.cc:729] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7b13c013f4460b3d59a4616dd17df, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.703367 30738 raft_consensus.cc:504] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.703890 30734 consensus_queue.cc:260] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.706380 30734 ts_tablet_manager.cc:1428] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260806 09:03:42.706456 30567 heartbeater.cc:503] Master 127.27.187.190:44691 was elected leader, sending a full tablet report...
I20260806 09:03:42.711467 30738 leader_election.cc:288] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:42.713295 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.706171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:40557 (local address 127.27.187.130:44083)
0806 09:03:42.706387 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.706394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.706419 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:03:42.706518 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.706526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.706625 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.706793 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:42.707754 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.708812 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.711404 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.712489 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.712675 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.712752 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.712983 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.713098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:03:42.713450 30597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.705718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:44083 (local address 127.27.187.131:40557)
0806 09:03:42.705978 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.706000 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.706150 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.707168 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.707176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.707216 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:42.707516 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.707531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.708994 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.709000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.711178 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.711201 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.712631 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.712637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.712796 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.712821 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.712940 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.713113 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 09:03:42.714673 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78024efd9c3948e99a1ef1e02bdfba11" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df" is_pre_election: true
I20260806 09:03:42.715394 30516 raft_consensus.cc:2454] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:42.716212 30590 leader_election.cc:302] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa7b13c013f4460b3d59a4616dd17df, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:42.716713 30738 raft_consensus.cc:2781] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:42.716903 30738 raft_consensus.cc:482] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:42.717022 30738 raft_consensus.cc:3037] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:42.718751 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.706342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:48441 (local address 127.27.187.129:46045)
0806 09:03:42.707056 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.707066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.712445 (+ 5379us) server_negotiation.cc:407] Connection header received
0806 09:03:42.712680 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.712686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.712781 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.712870 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:42.713633 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.714405 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.716818 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.717743 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.717878 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.717931 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.718372 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.718485 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573}
I20260806 09:03:42.718914 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.705834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46045 (local address 127.27.187.131:48441)
0806 09:03:42.712324 (+ 6490us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.712345 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.712502 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.712909 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.712915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.712945 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:42.713274 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.713288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.714550 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.714557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.716542 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.716557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.717971 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.717977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.718166 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.718202 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.718307 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.718468 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6346,"thread_start_us":6221,"threads_started":1}
I20260806 09:03:42.720001 30375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78024efd9c3948e99a1ef1e02bdfba11" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a" is_pre_election: true
I20260806 09:03:42.720077 30738 raft_consensus.cc:504] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.720599 30375 raft_consensus.cc:2454] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:42.720876 30738 leader_election.cc:288] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Requested vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:42.721407 30375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78024efd9c3948e99a1ef1e02bdfba11" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a"
I20260806 09:03:42.721450 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78024efd9c3948e99a1ef1e02bdfba11" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df"
I20260806 09:03:42.721642 30375 raft_consensus.cc:3037] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:42.721633 30516 raft_consensus.cc:3037] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:42.724634 30375 raft_consensus.cc:2454] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:42.724752 30516 raft_consensus.cc:2454] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:42.725155 30588 leader_election.cc:302] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:42.725505 30738 raft_consensus.cc:2781] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:42.725709 30738 raft_consensus.cc:686] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Becoming Leader. State: Replica: 9a80ec3302c84886aec940727d839808, State: Running, Role: LEADER
I20260806 09:03:42.726310 30738 consensus_queue.cc:237] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } }
I20260806 09:03:42.731714 30213 catalog_manager.cc:5495] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a80ec3302c84886aec940727d839808 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } health_report { overall_health: UNKNOWN } } }
W20260806 09:03:42.770088 30713 tablet.cc:2369] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:03:42 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007020, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:42.781232 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.765448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.27.187.190:44691)
0806 09:03:42.765744 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.765756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.766275 (+ 519us) server_negotiation.cc:407] Connection header received
0806 09:03:42.766485 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.766490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.766564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.766675 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:42.768047 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.768793 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.770082 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.770596 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.770764 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.774514 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:42.774519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:42.774523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:42.774561 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:42.777884 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.778959 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.778966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.778974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.779108 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:42.779540 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:42.779546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:42.779549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:42.779864 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:42.780030 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.780511 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.780885 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":89}
I20260806 09:03:42.781255 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.765470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44691 (local address 127.0.0.1:35494)
0806 09:03:42.766146 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.766163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.766363 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.766737 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.766742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.767591 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:42.767881 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.767891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.768943 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.768948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.769934 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.769945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.770828 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.770834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.770995 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.771701 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:42.771721 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:42.774281 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:42.778096 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.778113 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.778119 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.778796 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.779242 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:42.779247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:42.779253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:42.779417 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:42.780004 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:42.780012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:42.780219 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.780404 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.780743 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":367,"threads_started":1}
I20260806 09:03:42.787652 30213 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:35494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:03:42.788679 30213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:42.796180 30353 tablet_service.cc:1467] Processing CreateTablet for tablet c35351e38a334f2e89e5134d049bcd10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6deac455c4c4385bdbf1cbaffc89570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:42.796232 30495 tablet_service.cc:1467] Processing CreateTablet for tablet c35351e38a334f2e89e5134d049bcd10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6deac455c4c4385bdbf1cbaffc89570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:42.797145 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35351e38a334f2e89e5134d049bcd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.797165 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35351e38a334f2e89e5134d049bcd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.796943 30639 tablet_service.cc:1467] Processing CreateTablet for tablet c35351e38a334f2e89e5134d049bcd10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6deac455c4c4385bdbf1cbaffc89570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:03:42.797801 30639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35351e38a334f2e89e5134d049bcd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:42.803550 30732 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: Bootstrap starting.
I20260806 09:03:42.804787 30734 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: Bootstrap starting.
I20260806 09:03:42.805464 30732 tablet_bootstrap.cc:654] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.806178 30731 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: Bootstrap starting.
I20260806 09:03:42.806524 30734 tablet_bootstrap.cc:654] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.808274 30731 tablet_bootstrap.cc:654] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:42.808276 30732 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: No bootstrap required, opened a new log
I20260806 09:03:42.808437 30732 ts_tablet_manager.cc:1397] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:03:42.809376 30734 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: No bootstrap required, opened a new log
I20260806 09:03:42.809554 30734 ts_tablet_manager.cc:1397] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 09:03:42.809379 30732 raft_consensus.cc:348] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.809685 30732 raft_consensus.cc:374] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.809799 30732 raft_consensus.cc:729] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.810087 30732 consensus_queue.cc:260] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.810374 30734 raft_consensus.cc:348] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.810732 30734 raft_consensus.cc:374] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.810834 30734 raft_consensus.cc:729] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7b13c013f4460b3d59a4616dd17df, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.810968 30731 tablet_bootstrap.cc:492] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: No bootstrap required, opened a new log
I20260806 09:03:42.810774 30739 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:42.811120 30731 ts_tablet_manager.cc:1397] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 09:03:42.811187 30739 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:42.811138 30734 consensus_queue.cc:260] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.811713 30732 ts_tablet_manager.cc:1428] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:03:42.811810 30734 ts_tablet_manager.cc:1428] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260806 09:03:42.811655 30740 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:42.812107 30740 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:42.813405 30731 raft_consensus.cc:348] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.813680 30731 raft_consensus.cc:374] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:42.813781 30731 raft_consensus.cc:729] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a80ec3302c84886aec940727d839808, State: Initialized, Role: FOLLOWER
I20260806 09:03:42.814020 30731 consensus_queue.cc:260] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.814707 30731 ts_tablet_manager.cc:1428] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260806 09:03:42.814668 30738 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:42.815534 30738 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 09:03:42.865383 30426 tablet.cc:2369] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:42.924752 30738 raft_consensus.cc:482] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:42.925093 30738 raft_consensus.cc:504] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.925983 30738 leader_election.cc:288] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:42.926795 30375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35351e38a334f2e89e5134d049bcd10" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a" is_pre_election: true
I20260806 09:03:42.926900 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35351e38a334f2e89e5134d049bcd10" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df" is_pre_election: true
I20260806 09:03:42.927116 30375 raft_consensus.cc:2454] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:42.927393 30516 raft_consensus.cc:2454] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:42.927665 30588 leader_election.cc:302] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:42.928045 30738 raft_consensus.cc:2781] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:42.928161 30738 raft_consensus.cc:482] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:42.928274 30738 raft_consensus.cc:3037] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:42.931135 30738 raft_consensus.cc:504] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.931931 30738 leader_election.cc:288] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Requested vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:42.932619 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35351e38a334f2e89e5134d049bcd10" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df"
I20260806 09:03:42.932605 30375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35351e38a334f2e89e5134d049bcd10" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a"
I20260806 09:03:42.932902 30516 raft_consensus.cc:3037] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:42.932886 30375 raft_consensus.cc:3037] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:03:42.934517 30568 tablet.cc:2369] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:03:42.935938 30516 raft_consensus.cc:2454] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:42.936208 30375 raft_consensus.cc:2454] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:42.936539 30590 leader_election.cc:302] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa7b13c013f4460b3d59a4616dd17df, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:42.936885 30738 raft_consensus.cc:2781] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:42.937071 30738 raft_consensus.cc:686] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Becoming Leader. State: Replica: 9a80ec3302c84886aec940727d839808, State: Running, Role: LEADER
I20260806 09:03:42.937354 30738 consensus_queue.cc:237] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:42.939215 30737 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: TxnStatusTablet state changed. Reason: New leader 9a80ec3302c84886aec940727d839808. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:42.939647 30737 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:42.940462 30753 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:03:42.940784 30753 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:03:42.941174 30211 catalog_manager.cc:5495] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a80ec3302c84886aec940727d839808 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } health_report { overall_health: HEALTHY } } }
I20260806 09:03:42.974161 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.967861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42503 (local address 127.0.0.1:42364)
0806 09:03:42.968191 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:42.968210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:42.968344 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:42.968972 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:42.968977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:42.968989 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:42.969176 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.969187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.970267 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.970273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.971507 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:42.971521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.972368 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.972374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:42.972576 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.972668 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:42.973589 (+ 921us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:42.973616 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 09:03:42.973697 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 09:03:42.973855 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 09:03:42.974151 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:42.967842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.27.187.131:42503)
0806 09:03:42.968608 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:42.968613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:42.968629 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:03:42.968708 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:42.968711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:42.968785 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:42.968949 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:42.969332 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.970127 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.971667 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:42.972194 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:42.972343 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:42.972880 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:42.973458 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:42.973625 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:42.973733 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 09:03:42.973818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627}
I20260806 09:03:42.983331 30516 raft_consensus.cc:1261] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:42.983402 30375 raft_consensus.cc:1261] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:42.984401 30737 consensus_queue.cc:1035] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:42.985117 30738 consensus_queue.cc:1035] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:42.997617 30739 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: TxnStatusTablet state changed. Reason: New leader 9a80ec3302c84886aec940727d839808. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:42.998059 30739 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:43.001268 30757 mvcc.cc:205] Tried to move back new op lower bound from 7315484766129352704 to 7315484765953814528. Current Snapshot: MvccSnapshot[applied={T|T < 7315484766129352704}]
I20260806 09:03:43.007575 30740 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: TxnStatusTablet state changed. Reason: New leader 9a80ec3302c84886aec940727d839808. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:43.008103 30740 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:43.010558 30737 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:43.011005 30737 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:43.011411 30737 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:43.011741 30737 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 9a80ec3302c84886aec940727d839808: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:03:43.014034 30739 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:43.014343 30739 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 05bd647cd6054e6982f04d8d85246a0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:43.025225 30740 tablet_replica.cc:439] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } } }
I20260806 09:03:43.025525 30740 tablet_replica.cc:442] T c35351e38a334f2e89e5134d049bcd10 P 3aa7b13c013f4460b3d59a4616dd17df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:03:43.034562 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.028244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.27.187.131:42503)
0806 09:03:43.028496 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.028504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.028530 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:03:43.028696 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.028704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.028824 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.028959 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:43.029446 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.030515 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.032552 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.033039 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.033169 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.033746 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.033870 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.033949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.034116 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.034197 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:03:43.057077 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.047348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:45575 (local address 127.27.187.131:42503)
0806 09:03:43.047523 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.047527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.048332 (+ 805us) server_negotiation.cc:407] Connection header received
0806 09:03:43.048583 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.048586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.048647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.048751 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:43.049417 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.050327 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.054981 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.055468 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.055612 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.056148 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.056277 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.056381 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.056517 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.056707 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":52}
I20260806 09:03:43.057204 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.047437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42503 (local address 127.27.187.131:45575)
0806 09:03:43.048120 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:43.048135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:43.048428 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:43.048780 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:43.048784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:43.048795 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:43.049246 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.049258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.050466 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.050471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.054816 (+ 4345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.054837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.055696 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.055703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.055851 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.055936 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.056380 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.056393 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 09:03:43.056500 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:03:43.056763 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":82,"thread_start_us":399,"threads_started":1}
I20260806 09:03:43.068897 30375 raft_consensus.cc:1261] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:43.068970 30515 raft_consensus.cc:1261] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:43.069585 30737 consensus_queue.cc:1035] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:43.069953 30738 consensus_queue.cc:1035] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:43.081943 30756 mvcc.cc:205] Tried to move back new op lower bound from 7315484766483881984 to 7315484765092077568. Current Snapshot: MvccSnapshot[applied={T|T < 7315484766483881984}]
I20260806 09:03:43.178330 30212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:43.178719 30212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:43.184407 30212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Sending DeleteTablet for 3 replicas of tablet 78024efd9c3948e99a1ef1e02bdfba11
I20260806 09:03:43.186094 30639 tablet_service.cc:1514] Processing DeleteTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:43 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260806 09:03:43.186148 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:43 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53494
I20260806 09:03:43.187819 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet 78024efd9c3948e99a1ef1e02bdfba11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:43 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:55586
I20260806 09:03:43.189153 30774 tablet_replica.cc:330] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: stopping tablet replica
I20260806 09:03:43.189728 30774 raft_consensus.cc:2229] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:43.190246 30774 raft_consensus.cc:2258] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:43.191166 30212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:43.192217 30212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:43.195689 30773 tablet_replica.cc:330] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: stopping tablet replica
I20260806 09:03:43.196175 30773 raft_consensus.cc:2229] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:43.196678 30773 raft_consensus.cc:2258] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:43.199644 30772 tablet_replica.cc:330] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: stopping tablet replica
I20260806 09:03:43.199708 30773 ts_tablet_manager.cc:1905] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:43.200169 30772 raft_consensus.cc:2229] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:43.200979 30772 raft_consensus.cc:2258] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:43.203205 30772 ts_tablet_manager.cc:1905] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:43.203258 30638 tablet_service.cc:1467] Processing CreateTablet for tablet acf072d56e8942f6bd67ccee375a09c3 (DEFAULT_TABLE table=test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:43.203814 30638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf072d56e8942f6bd67ccee375a09c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:43.203943 30353 tablet_service.cc:1467] Processing CreateTablet for tablet acf072d56e8942f6bd67ccee375a09c3 (DEFAULT_TABLE table=test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:43.204030 30495 tablet_service.cc:1467] Processing CreateTablet for tablet acf072d56e8942f6bd67ccee375a09c3 (DEFAULT_TABLE table=test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:43.204599 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf072d56e8942f6bd67ccee375a09c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:43.204850 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf072d56e8942f6bd67ccee375a09c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:43.211416 30732 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Bootstrap starting.
I20260806 09:03:43.213406 30732 tablet_bootstrap.cc:654] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:43.213552 30734 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Bootstrap starting.
I20260806 09:03:43.215821 30734 tablet_bootstrap.cc:654] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:43.217931 30773 ts_tablet_manager.cc:1918] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:43.218128 30773 log.cc:1196] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/78024efd9c3948e99a1ef1e02bdfba11
I20260806 09:03:43.218724 30773 ts_tablet_manager.cc:1939] T 78024efd9c3948e99a1ef1e02bdfba11 P 05bd647cd6054e6982f04d8d85246a0a: Deleting consensus metadata
I20260806 09:03:43.220911 30198 catalog_manager.cc:4841] TS 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045): tablet 78024efd9c3948e99a1ef1e02bdfba11 (table test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]) successfully deleted
I20260806 09:03:43.222605 30734 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: No bootstrap required, opened a new log
I20260806 09:03:43.222862 30732 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: No bootstrap required, opened a new log
I20260806 09:03:43.222856 30734 ts_tablet_manager.cc:1397] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260806 09:03:43.223035 30732 ts_tablet_manager.cc:1397] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20260806 09:03:43.223775 30732 raft_consensus.cc:348] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.224064 30732 raft_consensus.cc:374] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:43.223871 30734 raft_consensus.cc:348] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.224150 30732 raft_consensus.cc:729] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Initialized, Role: FOLLOWER
I20260806 09:03:43.224196 30734 raft_consensus.cc:374] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:43.224320 30734 raft_consensus.cc:729] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7b13c013f4460b3d59a4616dd17df, State: Initialized, Role: FOLLOWER
I20260806 09:03:43.224471 30732 consensus_queue.cc:260] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.224676 30734 consensus_queue.cc:260] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.225787 30732 ts_tablet_manager.cc:1428] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 09:03:43.226152 30731 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Bootstrap starting.
I20260806 09:03:43.227447 30734 ts_tablet_manager.cc:1428] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260806 09:03:43.228050 30774 ts_tablet_manager.cc:1905] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:43.228118 30731 tablet_bootstrap.cc:654] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:43.233250 30772 ts_tablet_manager.cc:1918] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:43.233520 30772 log.cc:1196] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/78024efd9c3948e99a1ef1e02bdfba11
I20260806 09:03:43.234108 30772 ts_tablet_manager.cc:1939] T 78024efd9c3948e99a1ef1e02bdfba11 P 9a80ec3302c84886aec940727d839808: Deleting consensus metadata
I20260806 09:03:43.236481 30197 catalog_manager.cc:4841] TS 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503): tablet 78024efd9c3948e99a1ef1e02bdfba11 (table test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]) successfully deleted
I20260806 09:03:43.237471 30774 ts_tablet_manager.cc:1918] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:43.237690 30774 log.cc:1196] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/78024efd9c3948e99a1ef1e02bdfba11
I20260806 09:03:43.238330 30774 ts_tablet_manager.cc:1939] T 78024efd9c3948e99a1ef1e02bdfba11 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting consensus metadata
I20260806 09:03:43.238405 30731 tablet_bootstrap.cc:492] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: No bootstrap required, opened a new log
I20260806 09:03:43.238600 30731 ts_tablet_manager.cc:1397] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260806 09:03:43.239671 30731 raft_consensus.cc:348] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.240015 30731 raft_consensus.cc:374] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:43.240116 30731 raft_consensus.cc:729] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a80ec3302c84886aec940727d839808, State: Initialized, Role: FOLLOWER
I20260806 09:03:43.240661 30200 catalog_manager.cc:4841] TS 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083): tablet 78024efd9c3948e99a1ef1e02bdfba11 (table test-table [id=f487e0f8bd6443e8ac65dba7183e6e9a]) successfully deleted
I20260806 09:03:43.241266 30731 consensus_queue.cc:260] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.242060 30731 ts_tablet_manager.cc:1428] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:03:43.533711 30739 raft_consensus.cc:482] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:43.533983 30739 raft_consensus.cc:504] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.535972 30739 leader_election.cc:288] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083), 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503)
I20260806 09:03:43.556547 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.537597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:56205 (local address 127.27.187.131:42503)
0806 09:03:43.537936 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.537946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.548467 (+ 10521us) server_negotiation.cc:407] Connection header received
0806 09:03:43.548940 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.548947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.549024 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.549090 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:43.549872 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.551054 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.553699 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.555207 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.555343 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.555418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.556073 (+ 655us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.556238 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:03:43.557677 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.537017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42503 (local address 127.27.187.129:56205)
0806 09:03:43.548331 (+ 11314us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:43.548354 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:43.548775 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:43.549297 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:43.549301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:43.549329 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:43.549705 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.549717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.551258 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.551263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.553478 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.553506 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.555650 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.555657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.555870 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.555911 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:03:43.556994 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0806 09:03:43.557163 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11151,"mutex_wait_us":6,"thread_start_us":5000,"threads_started":1}
I20260806 09:03:43.558806 30659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf072d56e8942f6bd67ccee375a09c3" candidate_uuid: "05bd647cd6054e6982f04d8d85246a0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a80ec3302c84886aec940727d839808" is_pre_election: true
I20260806 09:03:43.559229 30659 raft_consensus.cc:2454] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05bd647cd6054e6982f04d8d85246a0a in term 0.
I20260806 09:03:43.559952 30303 leader_election.cc:302] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:43.560501 30739 raft_consensus.cc:2781] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:43.560623 30739 raft_consensus.cc:482] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:43.560736 30739 raft_consensus.cc:3037] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:43.561204 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.536292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:44083 (local address 127.27.187.129:35709)
0806 09:03:43.542509 (+ 6217us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:43.542544 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:43.542793 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:43.553036 (+ 10243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:43.553042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:43.553076 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:03:43.553556 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.553572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.555460 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.555466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.557789 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.557808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.560050 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.560057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.560259 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.560293 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:03:43.560395 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:03:43.560556 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6002,"thread_start_us":12189,"threads_started":1}
I20260806 09:03:43.562783 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.536516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:35709 (local address 127.27.187.130:44083)
0806 09:03:43.552601 (+ 16085us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.552609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.552682 (+ 73us) server_negotiation.cc:407] Connection header received
0806 09:03:43.552759 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.552765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.552852 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.553784 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:03:43.554004 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.555268 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.558389 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.559796 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.559953 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.560031 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.562227 (+ 2196us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.562398 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15948,"thread_start_us":15763,"threads_started":1}
I20260806 09:03:43.563604 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf072d56e8942f6bd67ccee375a09c3" candidate_uuid: "05bd647cd6054e6982f04d8d85246a0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df" is_pre_election: true
I20260806 09:03:43.563959 30515 raft_consensus.cc:2454] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05bd647cd6054e6982f04d8d85246a0a in term 0.
I20260806 09:03:43.564065 30739 raft_consensus.cc:504] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.565335 30659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf072d56e8942f6bd67ccee375a09c3" candidate_uuid: "05bd647cd6054e6982f04d8d85246a0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a80ec3302c84886aec940727d839808"
I20260806 09:03:43.565421 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf072d56e8942f6bd67ccee375a09c3" candidate_uuid: "05bd647cd6054e6982f04d8d85246a0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df"
I20260806 09:03:43.565618 30515 raft_consensus.cc:3037] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:43.565606 30659 raft_consensus.cc:3037] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:43.566836 30739 leader_election.cc:288] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [CANDIDATE]: Term 1 election: Requested vote from peers 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083), 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503)
I20260806 09:03:43.568781 30515 raft_consensus.cc:2454] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05bd647cd6054e6982f04d8d85246a0a in term 1.
I20260806 09:03:43.569082 30659 raft_consensus.cc:2454] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05bd647cd6054e6982f04d8d85246a0a in term 1.
I20260806 09:03:43.569321 30306 leader_election.cc:302] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 3aa7b13c013f4460b3d59a4616dd17df; no voters:
I20260806 09:03:43.569730 30739 raft_consensus.cc:2781] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:43.570204 30739 raft_consensus.cc:686] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 1 LEADER]: Becoming Leader. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Running, Role: LEADER
I20260806 09:03:43.570652 30739 consensus_queue.cc:237] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:43.575287 30211 catalog_manager.cc:5495] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129). New cstate: current_term: 1 leader_uuid: "05bd647cd6054e6982f04d8d85246a0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } health_report { overall_health: UNKNOWN } } }
I20260806 09:03:43.608906 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.601816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.27.187.129:46045)
0806 09:03:43.602505 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.602512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.602937 (+ 425us) server_negotiation.cc:407] Connection header received
0806 09:03:43.603206 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.603211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.603364 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.603563 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:43.604138 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.604992 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.606399 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.606892 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.607045 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.607543 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.608165 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.608345 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.608445 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.608537 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":441,"threads_started":1}
I20260806 09:03:43.629936 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.621062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:42637 (local address 127.27.187.131:42503)
0806 09:03:43.621358 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.621368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.622428 (+ 1060us) server_negotiation.cc:407] Connection header received
0806 09:03:43.622739 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.622747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.622847 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.622975 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:43.623714 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.624952 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.627360 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.628282 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.628468 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.629008 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.629161 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.629275 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.629441 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.629541 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 09:03:43.630470 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.621206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42503 (local address 127.27.187.129:42637)
0806 09:03:43.622301 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:43.622320 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:43.622524 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:43.622987 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:43.622993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:43.623007 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:43.623475 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.623488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.625185 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.625190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.627183 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.627199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.628517 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.628526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.628708 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.628779 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.629278 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.629295 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 09:03:43.629412 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:03:43.629595 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":610,"threads_started":1}
I20260806 09:03:43.640990 30515 raft_consensus.cc:1261] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Refusing update from remote peer 05bd647cd6054e6982f04d8d85246a0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:43.641327 30659 raft_consensus.cc:1261] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Refusing update from remote peer 05bd647cd6054e6982f04d8d85246a0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:43.641832 30781 consensus_queue.cc:1035] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:43.642232 30739 consensus_queue.cc:1035] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:43.660164 30758 mvcc.cc:205] Tried to move back new op lower bound from 7315484768825581568 to 7315484768550547456. Current Snapshot: MvccSnapshot[applied={T|T < 7315484768825581568}]
I20260806 09:03:43.702562 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.693719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:38005 (local address 127.27.187.129:46045)
0806 09:03:43.693914 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:43.693920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:43.695587 (+ 1667us) server_negotiation.cc:407] Connection header received
0806 09:03:43.695795 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:43.695801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:43.695907 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:43.696043 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:43.696710 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.697773 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.700186 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.700895 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.701000 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.701685 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.701847 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.701910 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:43.702141 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:03:43.702236 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:03:43.703819 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:43.693806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46045 (local address 127.27.187.131:38005)
0806 09:03:43.695447 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:43.695469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:43.695632 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:43.696050 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:43.696056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:43.696069 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:03:43.696533 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.696548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.697915 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.697941 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.699958 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:43.699978 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:43.701227 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:43.701233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:43.701431 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:43.701488 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:03:43.702001 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:03:43.702011 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:03:43.703163 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0806 09:03:43.703359 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":359,"threads_started":1}
W20260806 09:03:43.705538 30757 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:03:45.359411 30213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:45.359699 30213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:45.365706 30213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Sending DeleteTablet for 3 replicas of tablet acf072d56e8942f6bd67ccee375a09c3
I20260806 09:03:45.366864 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet acf072d56e8942f6bd67ccee375a09c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:45 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53494
I20260806 09:03:45.367141 30638 tablet_service.cc:1514] Processing DeleteTablet for tablet acf072d56e8942f6bd67ccee375a09c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:45 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260806 09:03:45.367384 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet acf072d56e8942f6bd67ccee375a09c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:45 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:55586
I20260806 09:03:45.367945 30820 tablet_replica.cc:330] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: stopping tablet replica
I20260806 09:03:45.368247 30820 raft_consensus.cc:2229] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:45.380928 30820 raft_consensus.cc:2258] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:45.382241 30821 tablet_replica.cc:330] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: stopping tablet replica
I20260806 09:03:45.382468 30821 raft_consensus.cc:2229] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:45.382625 30820 ts_tablet_manager.cc:1905] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:45.384119 30822 tablet_replica.cc:330] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: stopping tablet replica
I20260806 09:03:45.384357 30822 raft_consensus.cc:2229] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:45.384620 30822 raft_consensus.cc:2258] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:45.385012 30821 raft_consensus.cc:2258] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:45.386148 30822 ts_tablet_manager.cc:1905] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:45.387346 30821 ts_tablet_manager.cc:1905] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:45.391880 30820 ts_tablet_manager.cc:1918] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:45.392222 30820 log.cc:1196] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/acf072d56e8942f6bd67ccee375a09c3
I20260806 09:03:45.393020 30820 ts_tablet_manager.cc:1939] T acf072d56e8942f6bd67ccee375a09c3 P 05bd647cd6054e6982f04d8d85246a0a: Deleting consensus metadata
I20260806 09:03:45.394747 30198 catalog_manager.cc:4841] TS 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045): tablet acf072d56e8942f6bd67ccee375a09c3 (table test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]) successfully deleted
I20260806 09:03:45.397998 30821 ts_tablet_manager.cc:1918] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:45.398190 30821 log.cc:1196] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/acf072d56e8942f6bd67ccee375a09c3
I20260806 09:03:45.398669 30821 ts_tablet_manager.cc:1939] T acf072d56e8942f6bd67ccee375a09c3 P 9a80ec3302c84886aec940727d839808: Deleting consensus metadata
I20260806 09:03:45.399380 30822 ts_tablet_manager.cc:1918] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:45.399608 30822 log.cc:1196] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/acf072d56e8942f6bd67ccee375a09c3
I20260806 09:03:45.400204 30197 catalog_manager.cc:4841] TS 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503): tablet acf072d56e8942f6bd67ccee375a09c3 (table test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]) successfully deleted
I20260806 09:03:45.400378 30822 ts_tablet_manager.cc:1939] T acf072d56e8942f6bd67ccee375a09c3 P 3aa7b13c013f4460b3d59a4616dd17df: Deleting consensus metadata
I20260806 09:03:45.401429 30200 catalog_manager.cc:4841] TS 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083): tablet acf072d56e8942f6bd67ccee375a09c3 (table test-table [id=0a4b6519c3174ca4a5c291a9a84962b7]) successfully deleted
Aug 06 09:03:45 dist-test-slave-8tdl krb5kdc[30168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:03:45 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:45.438809 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:45.421559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.27.187.190:44691)
0806 09:03:45.422497 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:45.422502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:45.422526 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:03:45.422626 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:45.422630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:45.422801 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:45.422902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:45.424374 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:45.425203 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:45.426852 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:45.427281 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:45.427392 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:45.432080 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:45.432087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:45.432091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:45.432135 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:45.435503 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:45.436324 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:45.436328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:45.436331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:45.436538 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:45.436990 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:45.436994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:45.436996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:45.437423 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:45.437622 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:45.438023 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 09:03:45.438359 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":634,"threads_started":1}
I20260806 09:03:45.443048 30213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:03:45.443948 30213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:45.451543 30353 tablet_service.cc:1467] Processing CreateTablet for tablet f0497469fc254db0848acf9c44f2dcdd (DEFAULT_TABLE table=test-table [id=041e9fe20cb04e66b1f7f71a58caac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:45.451680 30495 tablet_service.cc:1467] Processing CreateTablet for tablet f0497469fc254db0848acf9c44f2dcdd (DEFAULT_TABLE table=test-table [id=041e9fe20cb04e66b1f7f71a58caac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:45.452171 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0497469fc254db0848acf9c44f2dcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:45.452201 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0497469fc254db0848acf9c44f2dcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:45.452391 30638 tablet_service.cc:1467] Processing CreateTablet for tablet f0497469fc254db0848acf9c44f2dcdd (DEFAULT_TABLE table=test-table [id=041e9fe20cb04e66b1f7f71a58caac77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:45.452852 30638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0497469fc254db0848acf9c44f2dcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:45.458648 30831 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Bootstrap starting.
I20260806 09:03:45.459836 30832 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Bootstrap starting.
I20260806 09:03:45.460841 30833 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Bootstrap starting.
I20260806 09:03:45.461437 30831 tablet_bootstrap.cc:654] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:45.461890 30832 tablet_bootstrap.cc:654] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:45.462582 30833 tablet_bootstrap.cc:654] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:45.464687 30831 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: No bootstrap required, opened a new log
I20260806 09:03:45.464787 30832 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: No bootstrap required, opened a new log
I20260806 09:03:45.464872 30831 ts_tablet_manager.cc:1397] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 09:03:45.464936 30832 ts_tablet_manager.cc:1397] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 09:03:45.465106 30833 tablet_bootstrap.cc:492] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: No bootstrap required, opened a new log
I20260806 09:03:45.465291 30833 ts_tablet_manager.cc:1397] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:03:45.465787 30832 raft_consensus.cc:348] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.465775 30831 raft_consensus.cc:348] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.466045 30832 raft_consensus.cc:374] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:45.466059 30831 raft_consensus.cc:374] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:45.466164 30832 raft_consensus.cc:729] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7b13c013f4460b3d59a4616dd17df, State: Initialized, Role: FOLLOWER
I20260806 09:03:45.466174 30831 raft_consensus.cc:729] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Initialized, Role: FOLLOWER
I20260806 09:03:45.466163 30833 raft_consensus.cc:348] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.466481 30833 raft_consensus.cc:374] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:45.466421 30832 consensus_queue.cc:260] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.466437 30831 consensus_queue.cc:260] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.466593 30833 raft_consensus.cc:729] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a80ec3302c84886aec940727d839808, State: Initialized, Role: FOLLOWER
I20260806 09:03:45.467048 30831 ts_tablet_manager.cc:1428] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 09:03:45.467530 30832 ts_tablet_manager.cc:1428] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:03:45.466969 30833 consensus_queue.cc:260] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.468886 30833 ts_tablet_manager.cc:1428] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:03:45.766548 30804 raft_consensus.cc:482] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:45.766922 30804 raft_consensus.cc:504] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.767954 30804 leader_election.cc:288] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:45.768672 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0497469fc254db0848acf9c44f2dcdd" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df" is_pre_election: true
I20260806 09:03:45.768790 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0497469fc254db0848acf9c44f2dcdd" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a" is_pre_election: true
I20260806 09:03:45.769114 30516 raft_consensus.cc:2454] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:45.769269 30374 raft_consensus.cc:2454] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:45.769879 30590 leader_election.cc:302] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa7b13c013f4460b3d59a4616dd17df, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:45.770361 30804 raft_consensus.cc:2781] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:45.770498 30804 raft_consensus.cc:482] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:45.770566 30804 raft_consensus.cc:3037] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:45.833461 30804 raft_consensus.cc:504] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.834595 30804 leader_election.cc:288] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Requested vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:45.835470 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0497469fc254db0848acf9c44f2dcdd" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df"
I20260806 09:03:45.835582 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0497469fc254db0848acf9c44f2dcdd" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a"
I20260806 09:03:45.835817 30516 raft_consensus.cc:3037] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:45.835875 30374 raft_consensus.cc:3037] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:45.904394 30516 raft_consensus.cc:2454] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:45.904417 30374 raft_consensus.cc:2454] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:45.905128 30588 leader_election.cc:302] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 3aa7b13c013f4460b3d59a4616dd17df, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:45.905613 30804 raft_consensus.cc:2781] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:45.906078 30804 raft_consensus.cc:686] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Becoming Leader. State: Replica: 9a80ec3302c84886aec940727d839808, State: Running, Role: LEADER
I20260806 09:03:45.906394 30804 consensus_queue.cc:237] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:45.910077 30212 catalog_manager.cc:5495] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a80ec3302c84886aec940727d839808 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } health_report { overall_health: HEALTHY } } }
I20260806 09:03:45.943989 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:45.935218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.27.187.131:42503)
0806 09:03:45.935869 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:45.935874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:45.935897 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:03:45.935963 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:45.935966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:45.936038 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:45.936140 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:03:45.936849 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:45.937911 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:45.940607 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:45.941665 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:45.941813 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:45.942965 (+ 1152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:03:45.943109 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:03:45.943288 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:45.943442 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 09:03:45.943526 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":381,"threads_started":1}
I20260806 09:03:45.957144 30516 raft_consensus.cc:1261] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:45.957908 30804 consensus_queue.cc:1035] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:45.965866 30374 raft_consensus.cc:1261] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:45.966826 30839 consensus_queue.cc:1035] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:46.032909 30211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:46.033156 30211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:46.037041 30211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Sending DeleteTablet for 3 replicas of tablet f0497469fc254db0848acf9c44f2dcdd
I20260806 09:03:46.038036 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet f0497469fc254db0848acf9c44f2dcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:46 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:55586
I20260806 09:03:46.038190 30638 tablet_service.cc:1514] Processing DeleteTablet for tablet f0497469fc254db0848acf9c44f2dcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:46 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260806 09:03:46.038581 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet f0497469fc254db0848acf9c44f2dcdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:46 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53494
I20260806 09:03:46.039074 30857 tablet_replica.cc:330] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: stopping tablet replica
I20260806 09:03:46.039322 30857 raft_consensus.cc:2229] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:46.039368 30858 tablet_replica.cc:330] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: stopping tablet replica
I20260806 09:03:46.039480 30859 tablet_replica.cc:330] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: stopping tablet replica
I20260806 09:03:46.039593 30857 raft_consensus.cc:2258] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:46.039770 30858 raft_consensus.cc:2229] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:46.039724 30859 raft_consensus.cc:2229] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:46.039979 30211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 09:03:46.040433 30859 raft_consensus.cc:2258] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:46.040694 30858 raft_consensus.cc:2258] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 09:03:46.040740 30211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:03:46.040892 30857 ts_tablet_manager.cc:1905] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:46.041724 30859 ts_tablet_manager.cc:1905] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:46.042026 30858 ts_tablet_manager.cc:1905] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:46.049225 30495 tablet_service.cc:1467] Processing CreateTablet for tablet 521ccdb95eda476191afe6c1e74c280c (DEFAULT_TABLE table=test-table [id=2395b25d869948aca1abd975f6b5b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:46.049232 30353 tablet_service.cc:1467] Processing CreateTablet for tablet 521ccdb95eda476191afe6c1e74c280c (DEFAULT_TABLE table=test-table [id=2395b25d869948aca1abd975f6b5b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:46.049482 30638 tablet_service.cc:1467] Processing CreateTablet for tablet 521ccdb95eda476191afe6c1e74c280c (DEFAULT_TABLE table=test-table [id=2395b25d869948aca1abd975f6b5b113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:03:46.105496 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521ccdb95eda476191afe6c1e74c280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:46.105504 30638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521ccdb95eda476191afe6c1e74c280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:46.109709 30858 ts_tablet_manager.cc:1918] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:46.109908 30858 log.cc:1196] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/f0497469fc254db0848acf9c44f2dcdd
I20260806 09:03:46.110406 30858 ts_tablet_manager.cc:1939] T f0497469fc254db0848acf9c44f2dcdd P 9a80ec3302c84886aec940727d839808: Deleting consensus metadata
I20260806 09:03:46.111979 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521ccdb95eda476191afe6c1e74c280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:46.118409 30861 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Bootstrap starting.
I20260806 09:03:46.118516 30860 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Bootstrap starting.
I20260806 09:03:46.120471 30861 tablet_bootstrap.cc:654] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:46.120910 30860 tablet_bootstrap.cc:654] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:46.121274 30862 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Bootstrap starting.
I20260806 09:03:46.123514 30861 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: No bootstrap required, opened a new log
I20260806 09:03:46.123759 30862 tablet_bootstrap.cc:654] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:46.123744 30861 ts_tablet_manager.cc:1397] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:03:46.124684 30861 raft_consensus.cc:348] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.125111 30861 raft_consensus.cc:374] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:46.125211 30860 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: No bootstrap required, opened a new log
I20260806 09:03:46.125268 30861 raft_consensus.cc:729] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7b13c013f4460b3d59a4616dd17df, State: Initialized, Role: FOLLOWER
I20260806 09:03:46.125399 30860 ts_tablet_manager.cc:1397] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 09:03:46.125584 30861 consensus_queue.cc:260] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.126339 30862 tablet_bootstrap.cc:492] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: No bootstrap required, opened a new log
I20260806 09:03:46.126333 30861 ts_tablet_manager.cc:1428] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:03:46.126490 30862 ts_tablet_manager.cc:1397] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 09:03:46.126339 30860 raft_consensus.cc:348] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.126629 30860 raft_consensus.cc:374] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:46.126778 30860 raft_consensus.cc:729] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a80ec3302c84886aec940727d839808, State: Initialized, Role: FOLLOWER
I20260806 09:03:46.127036 30860 consensus_queue.cc:260] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.127382 30862 raft_consensus.cc:348] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.127640 30862 raft_consensus.cc:374] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:46.127700 30860 ts_tablet_manager.cc:1428] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:03:46.127753 30862 raft_consensus.cc:729] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bd647cd6054e6982f04d8d85246a0a, State: Initialized, Role: FOLLOWER
I20260806 09:03:46.128044 30862 consensus_queue.cc:260] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.129352 30862 ts_tablet_manager.cc:1428] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 09:03:46.163851 30197 catalog_manager.cc:4841] TS 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503): tablet f0497469fc254db0848acf9c44f2dcdd (table test-table [id=041e9fe20cb04e66b1f7f71a58caac77]) successfully deleted
I20260806 09:03:46.202533 30859 ts_tablet_manager.cc:1918] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:46.202811 30859 log.cc:1196] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/f0497469fc254db0848acf9c44f2dcdd
I20260806 09:03:46.202539 30857 ts_tablet_manager.cc:1918] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:03:46.203193 30857 log.cc:1196] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/f0497469fc254db0848acf9c44f2dcdd
I20260806 09:03:46.203704 30859 ts_tablet_manager.cc:1939] T f0497469fc254db0848acf9c44f2dcdd P 05bd647cd6054e6982f04d8d85246a0a: Deleting consensus metadata
I20260806 09:03:46.204078 30857 ts_tablet_manager.cc:1939] T f0497469fc254db0848acf9c44f2dcdd P 3aa7b13c013f4460b3d59a4616dd17df: Deleting consensus metadata
I20260806 09:03:46.205355 30200 catalog_manager.cc:4841] TS 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083): tablet f0497469fc254db0848acf9c44f2dcdd (table test-table [id=041e9fe20cb04e66b1f7f71a58caac77]) successfully deleted
I20260806 09:03:46.205375 30198 catalog_manager.cc:4841] TS 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045): tablet f0497469fc254db0848acf9c44f2dcdd (table test-table [id=041e9fe20cb04e66b1f7f71a58caac77]) successfully deleted
I20260806 09:03:46.434490 30804 raft_consensus.cc:482] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:03:46.434995 30804 raft_consensus.cc:504] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.436169 30804 leader_election.cc:288] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:46.436691 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521ccdb95eda476191afe6c1e74c280c" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a" is_pre_election: true
I20260806 09:03:46.436800 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521ccdb95eda476191afe6c1e74c280c" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df" is_pre_election: true
I20260806 09:03:46.437158 30516 raft_consensus.cc:2454] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:46.437419 30374 raft_consensus.cc:2454] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 0.
I20260806 09:03:46.437654 30590 leader_election.cc:302] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa7b13c013f4460b3d59a4616dd17df, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:46.438016 30804 raft_consensus.cc:2781] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:03:46.438170 30804 raft_consensus.cc:482] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:03:46.438309 30804 raft_consensus.cc:3037] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:46.502952 30804 raft_consensus.cc:504] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.503849 30804 leader_election.cc:288] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Requested vote from peers 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045), 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083)
I20260806 09:03:46.504518 30374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521ccdb95eda476191afe6c1e74c280c" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bd647cd6054e6982f04d8d85246a0a"
I20260806 09:03:46.504536 30516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521ccdb95eda476191afe6c1e74c280c" candidate_uuid: "9a80ec3302c84886aec940727d839808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa7b13c013f4460b3d59a4616dd17df"
I20260806 09:03:46.504783 30374 raft_consensus.cc:3037] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:46.504828 30516 raft_consensus.cc:3037] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:46.507534 30374 raft_consensus.cc:2454] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:46.508122 30588 leader_election.cc:302] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bd647cd6054e6982f04d8d85246a0a, 9a80ec3302c84886aec940727d839808; no voters:
I20260806 09:03:46.508483 30804 raft_consensus.cc:2781] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:46.508549 30516 raft_consensus.cc:2454] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a80ec3302c84886aec940727d839808 in term 1.
I20260806 09:03:46.508701 30804 raft_consensus.cc:686] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Becoming Leader. State: Replica: 9a80ec3302c84886aec940727d839808, State: Running, Role: LEADER
I20260806 09:03:46.509378 30804 consensus_queue.cc:237] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } }
I20260806 09:03:46.512620 30213 catalog_manager.cc:5495] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a80ec3302c84886aec940727d839808 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "9a80ec3302c84886aec940727d839808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a80ec3302c84886aec940727d839808" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42503 } health_report { overall_health: HEALTHY } } }
I20260806 09:03:46.557929 30374 raft_consensus.cc:1261] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:46.558158 30516 raft_consensus.cc:1261] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Refusing update from remote peer 9a80ec3302c84886aec940727d839808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:03:46.558930 30804 consensus_queue.cc:1035] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa7b13c013f4460b3d59a4616dd17df" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:03:46.559508 30850 consensus_queue.cc:1035] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bd647cd6054e6982f04d8d85246a0a" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:03:46.586444 30867 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:03:48.194276 30213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:48.194582 30213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:03:48.200232 30213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0719b323a5c426fbcddff91ad72038b: Sending DeleteTablet for 3 replicas of tablet 521ccdb95eda476191afe6c1e74c280c
I20260806 09:03:48.201469 30353 tablet_service.cc:1514] Processing DeleteTablet for tablet 521ccdb95eda476191afe6c1e74c280c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:48 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53494
I20260806 09:03:48.201474 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet 521ccdb95eda476191afe6c1e74c280c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:48 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:55586
I20260806 09:03:48.201514 30638 tablet_service.cc:1514] Processing DeleteTablet for tablet 521ccdb95eda476191afe6c1e74c280c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:03:48 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260806 09:03:48.202378 30898 tablet_replica.cc:330] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: stopping tablet replica
I20260806 09:03:48.202661 30898 raft_consensus.cc:2229] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:03:48.203022 30900 tablet_replica.cc:330] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: stopping tablet replica
I20260806 09:03:48.203218 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30287
I20260806 09:03:48.203243 30898 raft_consensus.cc:2258] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:48.203549 30900 raft_consensus.cc:2229] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:48.203778 30899 tablet_replica.cc:330] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: stopping tablet replica
I20260806 09:03:48.203925 30900 raft_consensus.cc:2258] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:48.204345 30899 raft_consensus.cc:2229] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:03:48.204675 30899 raft_consensus.cc:2258] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:03:48.204916 30898 ts_tablet_manager.cc:1905] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:48.205367 30900 ts_tablet_manager.cc:1905] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:48.212538 30899 ts_tablet_manager.cc:1905] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:03:48.214345 30898 ts_tablet_manager.cc:1918] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:48.214608 30898 log.cc:1196] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/521ccdb95eda476191afe6c1e74c280c
I20260806 09:03:48.215348 30898 ts_tablet_manager.cc:1939] T 521ccdb95eda476191afe6c1e74c280c P 9a80ec3302c84886aec940727d839808: Deleting consensus metadata
I20260806 09:03:48.215931 30900 ts_tablet_manager.cc:1918] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:48.216216 30900 log.cc:1196] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/521ccdb95eda476191afe6c1e74c280c
I20260806 09:03:48.217068 30197 catalog_manager.cc:4841] TS 9a80ec3302c84886aec940727d839808 (127.27.187.131:42503): tablet 521ccdb95eda476191afe6c1e74c280c (table test-table [id=2395b25d869948aca1abd975f6b5b113]) successfully deleted
I20260806 09:03:48.217154 30900 ts_tablet_manager.cc:1939] T 521ccdb95eda476191afe6c1e74c280c P 3aa7b13c013f4460b3d59a4616dd17df: Deleting consensus metadata
I20260806 09:03:48.218592 30200 catalog_manager.cc:4841] TS 3aa7b13c013f4460b3d59a4616dd17df (127.27.187.130:44083): tablet 521ccdb95eda476191afe6c1e74c280c (table test-table [id=2395b25d869948aca1abd975f6b5b113]) successfully deleted
I20260806 09:03:48.221489 30899 ts_tablet_manager.cc:1918] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:03:48.221704 30899 log.cc:1196] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/521ccdb95eda476191afe6c1e74c280c
I20260806 09:03:48.222151 30899 ts_tablet_manager.cc:1939] T 521ccdb95eda476191afe6c1e74c280c P 05bd647cd6054e6982f04d8d85246a0a: Deleting consensus metadata
I20260806 09:03:48.223397 30198 catalog_manager.cc:4841] TS 05bd647cd6054e6982f04d8d85246a0a (127.27.187.129:46045): tablet 521ccdb95eda476191afe6c1e74c280c (table test-table [id=2395b25d869948aca1abd975f6b5b113]) successfully deleted
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:48.241568 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:48.205930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.27.187.129:46045)
0806 09:03:48.219220 (+ 13290us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:48.219225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:48.219243 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:03:48.219295 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:48.219302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:48.219377 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:48.219526 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:48.220942 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.221828 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.223350 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.223938 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.224054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:48.234375 (+ 10321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:48.234396 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:48.234414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:48.234462 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:48.238027 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.239062 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.239070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.239073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.239321 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.239894 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.239898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.239900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.240376 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:48.240607 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:48.240949 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 09:03:48.241257 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13157,"thread_start_us":12980,"threads_started":1}
W20260806 09:03:48.242494 30418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52420
W20260806 09:03:48.243328 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30287: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:48.243597 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 30287
I20260806 09:03:48.270593 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30429
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:03:48.289292 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:48.271464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.27.187.130:44083)
0806 09:03:48.272014 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:48.272018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:48.272036 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:03:48.272106 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:48.272110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:48.272176 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:48.272335 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:48.273885 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.274661 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.276320 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.276773 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.276892 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:48.282428 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:48.282444 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:48.282461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:48.282505 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:48.285894 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.286844 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.286859 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.286863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.287038 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.287532 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.287537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.287539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.287953 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:48.288270 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:48.288706 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 09:03:48.288923 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":301,"threads_started":1}
W20260806 09:03:48.290266 30562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56510
W20260806 09:03:48.291199 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30429: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:48.291396 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 30429
I20260806 09:03:48.314867 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30571
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:03:48.334368 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:48.315926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.27.187.131:42503)
0806 09:03:48.316690 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:48.316695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:48.316714 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:03:48.316774 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:48.316838 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:48.316959 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:48.317100 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:48.318617 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.319531 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.321004 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:48.321486 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:48.321645 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:48.327650 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:48.327670 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:48.327688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:48.327741 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:48.331166 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.331999 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.332008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.332011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.332173 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:48.332663 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:48.332669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:48.332672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:48.333057 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:48.333429 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:48.333713 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 09:03:48.333962 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":469,"threads_started":1}
W20260806 09:03:48.335177 30707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684
W20260806 09:03:48.335861 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30571: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:48.336026 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 30571
I20260806 09:03:48.366834 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30181
I20260806 09:03:48.368311 30254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:48.490759 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 30181
2026-08-06T09:03:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9725 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30909](info): set up 2 sockets
Aug 06 09:03:48 dist-test-slave-8tdl krb5kdc[30909](info): commencing operation
krb5kdc: starting...
W20260806 09:03:49.654302 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:49 dist-test-slave-8tdl krb5kdc[30909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:49.908433 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:39187
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:39187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:50.297454 30922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:50.297793 30922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:50.297885 30922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:50.305756 30922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:50.305902 30922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:50.305953 30922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:50.306008 30922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:50.306056 30922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:50.316726 30922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:39187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:39187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:50.318467 30922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:50.320169 30922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:50.334380 30930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:50.334441 30928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:50.334467 30927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:50.335737 30922 server_base.cc:1034] running on GCE node
I20260806 09:03:50.336695 30922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:50.338426 30922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:50.339713 30922 hybrid_clock.cc:648] HybridClock initialized: now 1786007030339653 us; error 47 us; skew 500 ppm
I20260806 09:03:50.340161 30922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:50 dist-test-slave-8tdl krb5kdc[30909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007030, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:50.343858 30922 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:50.345649 30922 webserver.cc:460] Webserver started at http://127.27.187.190:45029/ using document root <none> and password file <none>
I20260806 09:03:50.346418 30922 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:50.346534 30922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:50.346911 30922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:50.350175 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "dd09d0ddc69d41c48c0560e62ab14add"
format_stamp: "Formatted at 2026-08-06 09:03:50 on dist-test-slave-8tdl"
server_key: "61072e2a2a551882db4365714430e32b"
server_key_iv: "d4b6a5442e42ee18e607bf82cac468a9"
server_key_version: "encryptionkey@0"
I20260806 09:03:50.351034 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "dd09d0ddc69d41c48c0560e62ab14add"
format_stamp: "Formatted at 2026-08-06 09:03:50 on dist-test-slave-8tdl"
server_key: "61072e2a2a551882db4365714430e32b"
server_key_iv: "d4b6a5442e42ee18e607bf82cac468a9"
server_key_version: "encryptionkey@0"
I20260806 09:03:50.357306 30922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:03:50.361756 30937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:50.363672 30922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:50.363857 30922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "dd09d0ddc69d41c48c0560e62ab14add"
format_stamp: "Formatted at 2026-08-06 09:03:50 on dist-test-slave-8tdl"
server_key: "61072e2a2a551882db4365714430e32b"
server_key_iv: "d4b6a5442e42ee18e607bf82cac468a9"
server_key_version: "encryptionkey@0"
I20260806 09:03:50.364149 30922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:50.390170 30922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:50.396366 30922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:50.396670 30922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:50.418968 30922 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:39187
I20260806 09:03:50.418999 30990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:39187 every 8 connection(s)
I20260806 09:03:50.420583 30922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:50.424841 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:50.425721 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 30922
I20260806 09:03:50.425997 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:50.426736 28398 external_mini_cluster.cc:1428] Setting key 4b2d0400007f32a8f1694f5b6e1ac901
I20260806 09:03:50.440395 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add: Bootstrap starting.
Aug 06 09:03:50 dist-test-slave-8tdl krb5kdc[30909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:50.444878 30991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:50.446054 30991 log.cc:824] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:50.449920 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add: No bootstrap required, opened a new log
I20260806 09:03:50.454567 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:50.428997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.27.187.190:39187)
0806 09:03:50.429933 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:50.429948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:50.429990 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:03:50.431743 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:50.431768 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:50.432254 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:50.432617 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:50.434825 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:50.436315 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:50.438330 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:50.439310 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:50.439466 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:50.445008 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:50.445024 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:50.445127 (+ 103us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:50.445177 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:50.449342 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:50.450459 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:50.450466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:50.450473 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:50.450728 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:50.451262 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:50.451266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:50.451268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:50.451891 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:50.452255 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:50.452837 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 09:03:50.453384 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":441,"threads_started":1}
I20260806 09:03:50.459898 30991 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } }
I20260806 09:03:50.460323 30991 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:50.460472 30991 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd09d0ddc69d41c48c0560e62ab14add, State: Initialized, Role: FOLLOWER
I20260806 09:03:50.461119 30991 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } }
I20260806 09:03:50.461380 30991 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:50.461522 30991 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:50.461669 30991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:50.464540 30991 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } }
I20260806 09:03:50.465212 30991 leader_election.cc:302] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd09d0ddc69d41c48c0560e62ab14add; no voters:
I20260806 09:03:50.465832 30991 leader_election.cc:288] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:50.466125 30996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:50.466907 30996 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [term 1 LEADER]: Becoming Leader. State: Replica: dd09d0ddc69d41c48c0560e62ab14add, State: Running, Role: LEADER
I20260806 09:03:50.467495 30996 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } }
I20260806 09:03:50.468612 30991 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:50.470436 30998 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd09d0ddc69d41c48c0560e62ab14add. Latest consensus state: current_term: 1 leader_uuid: "dd09d0ddc69d41c48c0560e62ab14add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } } }
I20260806 09:03:50.470945 30998 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:50.472079 30997 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd09d0ddc69d41c48c0560e62ab14add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd09d0ddc69d41c48c0560e62ab14add" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39187 } } }
I20260806 09:03:50.472550 30997 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:50.473843 31003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:50.480427 31003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:50.497872 31003 catalog_manager.cc:1347] Generated new cluster ID: 3baf62c50fa3452580c5e1d3fe444830
I20260806 09:03:50.498034 31003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:50.528599 31003 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:50.530051 31003 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:50.542968 31003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd09d0ddc69d41c48c0560e62ab14add: Generated new TSK 0
I20260806 09:03:50.544051 31003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:50.608453 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:51.016482 31017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:51.016830 31017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:51.016932 31017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:51.025035 31017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:51.025177 31017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:51.025303 31017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:51.036051 31017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:51.037444 31017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:51.039039 31017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:51.055457 31023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:51.055474 31025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:51.055546 31022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:51.056258 31017 server_base.cc:1034] running on GCE node
I20260806 09:03:51.057035 31017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:51.058562 31017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:51.059830 31017 hybrid_clock.cc:648] HybridClock initialized: now 1786007031059747 us; error 88 us; skew 500 ppm
I20260806 09:03:51.060331 31017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:51 dist-test-slave-8tdl krb5kdc[30909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:51.064432 31017 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:51.067335 31017 webserver.cc:460] Webserver started at http://127.27.187.129:44243/ using document root <none> and password file <none>
I20260806 09:03:51.068575 31017 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:51.068905 31017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:51.069516 31017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:51.073518 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "2e4957cc72e74d25b924d3eb9cdfed92"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "3f846c4601734ef1839656fb7eca5940"
server_key_iv: "4ba88a3b24b0f9a983c48739c9977400"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.074647 31017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "2e4957cc72e74d25b924d3eb9cdfed92"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "3f846c4601734ef1839656fb7eca5940"
server_key_iv: "4ba88a3b24b0f9a983c48739c9977400"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.082048 31017 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:03:51.087191 31032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.089293 31017 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:51.089488 31017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "2e4957cc72e74d25b924d3eb9cdfed92"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "3f846c4601734ef1839656fb7eca5940"
server_key_iv: "4ba88a3b24b0f9a983c48739c9977400"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.089699 31017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:51.117980 31017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:51.123219 31017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:51.123498 31017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:51.124532 31017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:51.126163 31017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:51.126248 31017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.126341 31017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:51.126384 31017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.164080 31017 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:38953
I20260806 09:03:51.164259 31148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:38953 every 8 connection(s)
I20260806 09:03:51.165469 31017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:51.166548 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31017
I20260806 09:03:51.166812 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:51.167690 28398 external_mini_cluster.cc:1428] Setting key 15ae466c2b5964dba9bc7cd154e0736a
Aug 06 09:03:51 dist-test-slave-8tdl krb5kdc[30909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:03:51.192397 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:51.168685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:33521 (local address 127.27.187.190:39187)
0806 09:03:51.169231 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:51.169237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:51.170664 (+ 1427us) server_negotiation.cc:407] Connection header received
0806 09:03:51.172706 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:51.172713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:51.172816 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:51.172939 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:51.174777 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.175673 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.177914 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.178385 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.178517 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:51.184110 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:51.184116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:51.184120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:51.184160 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:51.188030 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:51.189140 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:51.189145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:51.189148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:51.189302 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:51.189893 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:51.189898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:51.189900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:51.190274 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:51.190466 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:51.191681 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0806 09:03:51.191966 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":266,"threads_started":1}
I20260806 09:03:51.193842 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:51.169463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:39187 (local address 127.27.187.129:33521)
0806 09:03:51.170441 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:51.170550 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:51.172397 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:51.173226 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:51.173235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:51.173817 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:51.174509 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:51.174586 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.175856 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.175860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:51.177643 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:51.177669 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.179041 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.179052 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:51.179456 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:51.180461 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:51.180500 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:51.183777 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:51.188298 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:51.188308 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:51.188325 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:51.188932 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:51.189448 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:51.189453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:51.189470 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:51.189728 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:51.190447 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:51.190457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:51.191155 (+ 698us) client_negotiation.cc:769] Sending connection context
0806 09:03:51.191531 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 09:03:51.192207 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":417,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:03:51.196353 31149 heartbeater.cc:347] Connected to a master server at 127.27.187.190:39187
I20260806 09:03:51.196769 31149 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:51.197803 31149 heartbeater.cc:511] Master 127.27.187.190:39187 requested a full tablet report, sending...
I20260806 09:03:51.200789 30954 ts_manager.cc:194] Registered new tserver with Master: 2e4957cc72e74d25b924d3eb9cdfed92 (127.27.187.129:38953)
I20260806 09:03:51.203181 30954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:33521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:51.228288 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:51.593640 31155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:51.593994 31155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:51.594128 31155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:51.601878 31155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:51.602005 31155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:51.602180 31155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:51.613754 31155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:51.615463 31155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:51.617012 31155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:51.632725 31160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:51.632788 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:51.632648 31161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:51.636754 31155 server_base.cc:1034] running on GCE node
I20260806 09:03:51.637549 31155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:51.638908 31155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:51.640148 31155 hybrid_clock.cc:648] HybridClock initialized: now 1786007031640103 us; error 42 us; skew 500 ppm
I20260806 09:03:51.640581 31155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:51 dist-test-slave-8tdl krb5kdc[30909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:51.644667 31155 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:51.646471 31155 webserver.cc:460] Webserver started at http://127.27.187.130:45473/ using document root <none> and password file <none>
I20260806 09:03:51.647339 31155 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:51.647476 31155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:51.647825 31155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:51.651513 31155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "0c91a65eaacd45f789d8fcf7bf6489d6"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "11b3d5399b80893a8386ba94d65e37eb"
server_key_iv: "cefc3fda5e1c904a1cd45060ec781293"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.652462 31155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "0c91a65eaacd45f789d8fcf7bf6489d6"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "11b3d5399b80893a8386ba94d65e37eb"
server_key_iv: "cefc3fda5e1c904a1cd45060ec781293"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.658591 31155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:03:51.662951 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.664693 31155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:03:51.664909 31155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "0c91a65eaacd45f789d8fcf7bf6489d6"
format_stamp: "Formatted at 2026-08-06 09:03:51 on dist-test-slave-8tdl"
server_key: "11b3d5399b80893a8386ba94d65e37eb"
server_key_iv: "cefc3fda5e1c904a1cd45060ec781293"
server_key_version: "encryptionkey@0"
I20260806 09:03:51.665156 31155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:51.683698 31155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:51.688529 31155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:51.688854 31155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:51.689985 31155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:51.691756 31155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:51.691839 31155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.692018 31155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:51.692085 31155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:51.734993 31155 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:35669
I20260806 09:03:51.735060 31287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:35669 every 8 connection(s)
I20260806 09:03:51.736579 31155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:51.744556 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31155
I20260806 09:03:51.744789 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:51.745254 28398 external_mini_cluster.cc:1428] Setting key 3b99ff13b1aaa310a9ac90befc741dc1
Aug 06 09:03:51 dist-test-slave-8tdl krb5kdc[30909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:51.760681 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:51.740025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:56969 (local address 127.27.187.190:39187)
0806 09:03:51.740516 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:51.740521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:51.741554 (+ 1033us) server_negotiation.cc:407] Connection header received
0806 09:03:51.743680 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:51.743685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:51.743764 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:51.743870 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:51.745747 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.746519 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.748321 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.748772 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.748913 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:51.753378 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:51.753385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:51.753388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:51.753438 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:51.756784 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:51.757981 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:51.757986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:51.757988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:51.758129 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:51.758771 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:51.758775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:51.758777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:51.759072 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:51.759189 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:51.760207 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0806 09:03:51.760403 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260806 09:03:51.762343 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:51.740627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:39187 (local address 127.27.187.130:56969)
0806 09:03:51.741371 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:51.741412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:51.743314 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:51.744146 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:51.744155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:51.744832 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:51.745544 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:51.745565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.746747 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.746751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:51.748159 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:51.748172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:51.749042 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:51.749050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:51.749301 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:51.750187 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:51.750224 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:51.753042 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:51.757006 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:51.757020 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:51.757044 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:51.757801 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:51.758252 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:51.758260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:51.758266 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:51.758567 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:51.759216 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:51.759227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:51.759759 (+ 532us) client_negotiation.cc:769] Sending connection context
0806 09:03:51.760087 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 09:03:51.760736 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":282,"threads_started":1}
I20260806 09:03:51.764230 31288 heartbeater.cc:347] Connected to a master server at 127.27.187.190:39187
I20260806 09:03:51.764580 31288 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:51.765448 31288 heartbeater.cc:511] Master 127.27.187.190:39187 requested a full tablet report, sending...
I20260806 09:03:51.767423 30954 ts_manager.cc:194] Registered new tserver with Master: 0c91a65eaacd45f789d8fcf7bf6489d6 (127.27.187.130:35669)
I20260806 09:03:51.768326 30954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:56969
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:51.807296 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:52.188735 31294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:52.189261 31294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:52.189393 31294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:52.199396 31294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:52.199554 31294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:52.199710 31294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:52.206920 31149 heartbeater.cc:503] Master 127.27.187.190:39187 was elected leader, sending a full tablet report...
I20260806 09:03:52.212420 31294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:52.213891 31294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:52.215358 31294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:52.230314 31299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:52.230232 31300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:52.230283 31302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:52.231679 31294 server_base.cc:1034] running on GCE node
I20260806 09:03:52.232460 31294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:52.233590 31294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:52.234853 31294 hybrid_clock.cc:648] HybridClock initialized: now 1786007032234787 us; error 46 us; skew 500 ppm
I20260806 09:03:52.235239 31294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:52 dist-test-slave-8tdl krb5kdc[30909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007032, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:52.239295 31294 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:52.241406 31294 webserver.cc:460] Webserver started at http://127.27.187.131:38175/ using document root <none> and password file <none>
I20260806 09:03:52.242286 31294 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:52.242445 31294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:52.242900 31294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:52.246557 31294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "8280963afad848fda823e78c58fad10b"
format_stamp: "Formatted at 2026-08-06 09:03:52 on dist-test-slave-8tdl"
server_key: "dbed820658c743a4f5faa6a5cc66042d"
server_key_iv: "c0beab092946470441528dd2140c8152"
server_key_version: "encryptionkey@0"
I20260806 09:03:52.247655 31294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "8280963afad848fda823e78c58fad10b"
format_stamp: "Formatted at 2026-08-06 09:03:52 on dist-test-slave-8tdl"
server_key: "dbed820658c743a4f5faa6a5cc66042d"
server_key_iv: "c0beab092946470441528dd2140c8152"
server_key_version: "encryptionkey@0"
I20260806 09:03:52.254022 31294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:03:52.258870 31309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:52.260600 31294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:52.260833 31294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "8280963afad848fda823e78c58fad10b"
format_stamp: "Formatted at 2026-08-06 09:03:52 on dist-test-slave-8tdl"
server_key: "dbed820658c743a4f5faa6a5cc66042d"
server_key_iv: "c0beab092946470441528dd2140c8152"
server_key_version: "encryptionkey@0"
I20260806 09:03:52.261113 31294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:52.286290 31294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:52.291513 31294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:52.291783 31294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:52.293004 31294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:52.295068 31294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:52.295147 31294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:52.295244 31294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:52.295288 31294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:52.333339 31294 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:44813
I20260806 09:03:52.333444 31426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:44813 every 8 connection(s)
I20260806 09:03:52.334824 31294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:52.343482 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31294
I20260806 09:03:52.343703 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:52.344193 28398 external_mini_cluster.cc:1428] Setting key f1c7a82c72ed698edfd08c8fe64c2e07
Aug 06 09:03:52 dist-test-slave-8tdl krb5kdc[30909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007032, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:52.357857 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:52.337842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:45001 (local address 127.27.187.190:39187)
0806 09:03:52.338494 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:52.338499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:52.339391 (+ 892us) server_negotiation.cc:407] Connection header received
0806 09:03:52.341493 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:52.341501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:52.341617 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:52.341745 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:52.343538 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:52.344502 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:52.346491 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:52.347014 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:52.347171 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:52.351354 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:52.351360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:52.351363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:52.351401 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:52.354282 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:52.355475 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:52.355480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:52.355482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:52.355640 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:52.356124 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:52.356133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:52.356135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:52.356428 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:52.356552 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:52.357344 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 09:03:52.357550 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":402,"threads_started":1}
I20260806 09:03:52.359081 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:52.338327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:39187 (local address 127.27.187.131:45001)
0806 09:03:52.339193 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:52.339241 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:52.341169 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:52.342036 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:52.342046 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:52.342631 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:52.343337 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:52.343357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:52.344685 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:52.344712 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:52.346317 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:52.346338 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:52.347270 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:52.347279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:52.347610 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:52.348442 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:52.348467 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:52.351046 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:52.354471 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:52.354479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:52.354495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:52.355319 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:52.355775 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:52.355779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:52.355781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:52.355991 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:52.356556 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:52.356563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:52.356956 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 09:03:52.357211 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 09:03:52.357672 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260806 09:03:52.360705 31427 heartbeater.cc:347] Connected to a master server at 127.27.187.190:39187
I20260806 09:03:52.361074 31427 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:52.361861 31427 heartbeater.cc:511] Master 127.27.187.190:39187 requested a full tablet report, sending...
I20260806 09:03:52.363996 30953 ts_manager.cc:194] Registered new tserver with Master: 8280963afad848fda823e78c58fad10b (127.27.187.131:44813)
I20260806 09:03:52.365126 30953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:45001
I20260806 09:03:52.374053 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:03:52.417263 31429 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:03:52.414272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.27.187.190:39187)
0806 09:03:52.414533 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:52.414539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:52.414804 (+ 265us) server_negotiation.cc:407] Connection header received
0806 09:03:52.414983 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:52.414987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:52.415062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:52.415156 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:52.416889 (+ 1733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37108: BlockingRecv error: recv got EOF from 127.0.0.1:37108 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:03:52.422272 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31017
W20260806 09:03:52.429100 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31017: Not authorized: Client connection negotiation failed: client connection to 127.27.187.129:38953: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:03:52.429328 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31017
I20260806 09:03:52.447602 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31155
W20260806 09:03:52.451107 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31155: Not authorized: Client connection negotiation failed: client connection to 127.27.187.130:35669: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:03:52.451376 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31155
I20260806 09:03:52.470413 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31294
W20260806 09:03:52.474195 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31294: Not authorized: Client connection negotiation failed: client connection to 127.27.187.131:44813: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:03:52.474421 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31294
I20260806 09:03:52.494060 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 30922
I20260806 09:03:52.495697 30986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:52.613436 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 30922
2026-08-06T09:03:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4121 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:52 dist-test-slave-8tdl krb5kdc[31445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:52 dist-test-slave-8tdl krb5kdc[31445](info): set up 2 sockets
Aug 06 09:03:52 dist-test-slave-8tdl krb5kdc[31445](info): commencing operation
krb5kdc: starting...
W20260806 09:03:53.802255 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:53 dist-test-slave-8tdl krb5kdc[31445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:54.080436 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:41301
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:41301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:54.439889 31458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:54.440232 31458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:54.440380 31458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:54.448134 31458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:54.448271 31458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:54.448371 31458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:54.448475 31458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:54.448582 31458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:54.459368 31458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:41301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:41301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:54.460867 31458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:54.462419 31458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:54.476470 31463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:54.476608 31464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:54.477422 31466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:54.477615 31458 server_base.cc:1034] running on GCE node
I20260806 09:03:54.478668 31458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:54.480655 31458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:54.482160 31458 hybrid_clock.cc:648] HybridClock initialized: now 1786007034482071 us; error 69 us; skew 500 ppm
I20260806 09:03:54.482662 31458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:54 dist-test-slave-8tdl krb5kdc[31445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:54.487429 31458 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:54.489532 31458 webserver.cc:460] Webserver started at http://127.27.187.190:38189/ using document root <none> and password file <none>
I20260806 09:03:54.490397 31458 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:54.490538 31458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:54.490935 31458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:54.494607 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "438f43ddc77f47dc88f3c26ae752aa11"
format_stamp: "Formatted at 2026-08-06 09:03:54 on dist-test-slave-8tdl"
server_key: "82f574acc7cadf552e5b158d498883de"
server_key_iv: "7ac05d27540fd317bce3bfdbf8872154"
server_key_version: "encryptionkey@0"
I20260806 09:03:54.496464 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "438f43ddc77f47dc88f3c26ae752aa11"
format_stamp: "Formatted at 2026-08-06 09:03:54 on dist-test-slave-8tdl"
server_key: "82f574acc7cadf552e5b158d498883de"
server_key_iv: "7ac05d27540fd317bce3bfdbf8872154"
server_key_version: "encryptionkey@0"
I20260806 09:03:54.503450 31458 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260806 09:03:54.508914 31473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:54.510933 31458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:54.511144 31458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "438f43ddc77f47dc88f3c26ae752aa11"
format_stamp: "Formatted at 2026-08-06 09:03:54 on dist-test-slave-8tdl"
server_key: "82f574acc7cadf552e5b158d498883de"
server_key_iv: "7ac05d27540fd317bce3bfdbf8872154"
server_key_version: "encryptionkey@0"
I20260806 09:03:54.511448 31458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:54.564821 31458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:54.571156 31458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:54.571544 31458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:54.594638 31458 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:41301
I20260806 09:03:54.594779 31525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:41301 every 8 connection(s)
I20260806 09:03:54.596481 31458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:54.598456 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31458
I20260806 09:03:54.598770 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:54.599711 28398 external_mini_cluster.cc:1428] Setting key a8df5e86ede0f57f04713fa763a2a9f4
I20260806 09:03:54.602159 31526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:03:54 dist-test-slave-8tdl krb5kdc[31445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:54.617705 31526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11: Bootstrap starting.
I20260806 09:03:54.621488 31526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:54.622763 31526 log.cc:824] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:54.626046 31526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11: No bootstrap required, opened a new log
I20260806 09:03:54.626380 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:54.601867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.27.187.190:41301)
0806 09:03:54.602831 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:54.602844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:54.602891 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:03:54.604933 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:54.604962 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:54.605455 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:54.605935 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:54.608449 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:54.610055 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:54.612056 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:54.612831 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:54.613002 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:54.618305 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:54.618315 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:54.618331 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:54.618381 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:54.621801 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:54.622634 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:54.622641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:54.622648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:54.622960 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:54.623392 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:54.623401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:54.623404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:54.624011 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:54.624348 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:54.624819 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 09:03:54.625287 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":444,"threads_started":1}
I20260806 09:03:54.630560 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:03:54.634421 31526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } }
I20260806 09:03:54.634999 31526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:54.635121 31526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438f43ddc77f47dc88f3c26ae752aa11, State: Initialized, Role: FOLLOWER
I20260806 09:03:54.635823 31526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } }
I20260806 09:03:54.636085 31526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:54.636250 31526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:54.636442 31526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:54.638893 31526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } }
I20260806 09:03:54.639484 31526 leader_election.cc:302] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 438f43ddc77f47dc88f3c26ae752aa11; no voters:
I20260806 09:03:54.640267 31526 leader_election.cc:288] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:54.640383 31533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:54.643272 31526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:54.647054 31533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [term 1 LEADER]: Becoming Leader. State: Replica: 438f43ddc77f47dc88f3c26ae752aa11, State: Running, Role: LEADER
I20260806 09:03:54.647958 31533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } }
I20260806 09:03:54.652751 31535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438f43ddc77f47dc88f3c26ae752aa11. Latest consensus state: current_term: 1 leader_uuid: "438f43ddc77f47dc88f3c26ae752aa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } } }
I20260806 09:03:54.653442 31535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:54.652880 31534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "438f43ddc77f47dc88f3c26ae752aa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438f43ddc77f47dc88f3c26ae752aa11" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 41301 } } }
I20260806 09:03:54.654234 31534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:54.662948 31542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:54.670149 31542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:54.683552 31551 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:03:54.686836 31542 catalog_manager.cc:1347] Generated new cluster ID: 8a6a5a63ca3d4d058ebb4b9f47422208
I20260806 09:03:54.686982 31542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:54.700306 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:54.687811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.27.187.190:41301)
0806 09:03:54.688019 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:54.688025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:54.690341 (+ 2316us) server_negotiation.cc:407] Connection header received
0806 09:03:54.690531 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:54.690537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:54.690645 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:54.690795 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:54.692140 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:54.693423 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:54.695598 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:54.696368 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:54.696538 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:54.697974 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:54.697981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:54.697986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:03:54.699160 (+ 1174us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:54.699233 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:03:54.699271 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:54.699401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:54.699630 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 09:03:54.699748 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":71}
I20260806 09:03:54.700598 31542 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:54.702648 31542 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:54.717176 31542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 438f43ddc77f47dc88f3c26ae752aa11: Generated new TSK 0
I20260806 09:03:54.718482 31542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:03:54.732487 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31458
I20260806 09:03:54.734062 31521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:54.829793 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31458
2026-08-06T09:03:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2209 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:54 dist-test-slave-8tdl krb5kdc[31562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:54 dist-test-slave-8tdl krb5kdc[31562](info): set up 2 sockets
Aug 06 09:03:54 dist-test-slave-8tdl krb5kdc[31562](info): commencing operation
krb5kdc: starting...
W20260806 09:03:55.974040 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:03:56 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:03:56.227964 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:42593
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:56.581718 31575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:56.582063 31575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:56.582247 31575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:56.590667 31575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:03:56.590843 31575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:56.590978 31575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:56.591097 31575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:03:56.591182 31575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:03:56.601675 31575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:42593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:56.603225 31575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:56.604631 31575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:56.617823 31581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:56.617853 31582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:56.617930 31584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:56.618810 31575 server_base.cc:1034] running on GCE node
I20260806 09:03:56.619753 31575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:56.621462 31575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:56.622821 31575 hybrid_clock.cc:648] HybridClock initialized: now 1786007036622779 us; error 66 us; skew 500 ppm
I20260806 09:03:56.623174 31575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:56 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007036, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:56.626947 31575 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:03:56.628814 31575 webserver.cc:460] Webserver started at http://127.27.187.190:34039/ using document root <none> and password file <none>
I20260806 09:03:56.629621 31575 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:56.629782 31575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:56.630154 31575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:56.633790 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "74f0d30af738433081c6d36f4bd99ba3"
format_stamp: "Formatted at 2026-08-06 09:03:56 on dist-test-slave-8tdl"
server_key: "b6d903af8cff7bd51f9002f028b70358"
server_key_iv: "c8371f8df9dd57e7dd4a0fca7fe27846"
server_key_version: "encryptionkey@0"
I20260806 09:03:56.634801 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "74f0d30af738433081c6d36f4bd99ba3"
format_stamp: "Formatted at 2026-08-06 09:03:56 on dist-test-slave-8tdl"
server_key: "b6d903af8cff7bd51f9002f028b70358"
server_key_iv: "c8371f8df9dd57e7dd4a0fca7fe27846"
server_key_version: "encryptionkey@0"
I20260806 09:03:56.641220 31575 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 09:03:56.645447 31591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:56.647341 31575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:03:56.647578 31575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "74f0d30af738433081c6d36f4bd99ba3"
format_stamp: "Formatted at 2026-08-06 09:03:56 on dist-test-slave-8tdl"
server_key: "b6d903af8cff7bd51f9002f028b70358"
server_key_iv: "c8371f8df9dd57e7dd4a0fca7fe27846"
server_key_version: "encryptionkey@0"
I20260806 09:03:56.647812 31575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:56.671702 31575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:56.676798 31575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:56.677124 31575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:56.697934 31575 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:42593
I20260806 09:03:56.697989 31643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:42593 every 8 connection(s)
I20260806 09:03:56.699430 31575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:03:56.703593 31644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:03:56.703787 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31575
I20260806 09:03:56.704005 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:03:56.704856 28398 external_mini_cluster.cc:1428] Setting key 9cf32985a6d551ff35ba28da029d2972
I20260806 09:03:56.719264 31644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3: Bootstrap starting.
Aug 06 09:03:56 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:56.723450 31644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3: Neither blocks nor log segments found. Creating new log.
I20260806 09:03:56.724642 31644 log.cc:824] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:03:56.728515 31644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3: No bootstrap required, opened a new log
I20260806 09:03:56.732542 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:56.707255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.27.187.190:42593)
0806 09:03:56.708130 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:56.708143 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:56.708190 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:03:56.710439 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:56.710467 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:56.711042 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:56.711431 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:56.713421 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:56.714914 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:56.716826 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:56.717735 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:56.717891 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:56.723163 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:56.723172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:56.723188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:56.723235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:56.727266 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:56.728127 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:56.728136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:56.728144 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:56.728370 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:56.728937 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:56.728942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:56.728945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:56.729751 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:56.730102 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:56.730469 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 09:03:56.731043 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":420,"threads_started":1}
I20260806 09:03:56.737823 31644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } }
I20260806 09:03:56.738282 31644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:03:56.738413 31644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0d30af738433081c6d36f4bd99ba3, State: Initialized, Role: FOLLOWER
I20260806 09:03:56.739289 31644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } }
I20260806 09:03:56.739594 31644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:03:56.739765 31644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:03:56.739991 31644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:03:56.742396 31644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } }
I20260806 09:03:56.743075 31644 leader_election.cc:302] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f0d30af738433081c6d36f4bd99ba3; no voters:
I20260806 09:03:56.743817 31644 leader_election.cc:288] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:03:56.743968 31649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:03:56.744746 31649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 74f0d30af738433081c6d36f4bd99ba3, State: Running, Role: LEADER
I20260806 09:03:56.745497 31649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } }
I20260806 09:03:56.746500 31644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:03:56.748555 31651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f0d30af738433081c6d36f4bd99ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } } }
I20260806 09:03:56.749030 31651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:56.750240 31656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:03:56.750181 31650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f0d30af738433081c6d36f4bd99ba3. Latest consensus state: current_term: 1 leader_uuid: "74f0d30af738433081c6d36f4bd99ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f0d30af738433081c6d36f4bd99ba3" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42593 } } }
I20260806 09:03:56.750756 31650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:03:56.757943 31656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:03:56.768730 31656 catalog_manager.cc:1347] Generated new cluster ID: 7c9bccbf482e45c5bb742f49c7fe55c8
I20260806 09:03:56.768939 31656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:03:56.787846 31656 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:03:56.789949 31656 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:03:56.800756 31656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f0d30af738433081c6d36f4bd99ba3: Generated new TSK 0
I20260806 09:03:56.801962 31656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:03:56.869231 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:57.244509 31670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:57.244915 31670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:57.245101 31670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:57.254099 31670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:57.254299 31670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:57.254506 31670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:03:57.266181 31670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:57.267755 31670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:57.269224 31670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:57.285362 31676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:57.285310 31678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:57.285408 31675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:57.286486 31670 server_base.cc:1034] running on GCE node
I20260806 09:03:57.287387 31670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:57.288501 31670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:57.289804 31670 hybrid_clock.cc:648] HybridClock initialized: now 1786007037289752 us; error 46 us; skew 500 ppm
I20260806 09:03:57.290279 31670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:57 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007037, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:57.294173 31670 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:03:57.295940 31670 webserver.cc:460] Webserver started at http://127.27.187.129:44927/ using document root <none> and password file <none>
I20260806 09:03:57.296715 31670 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:57.296878 31670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:57.297255 31670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:57.300976 31670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "555596133af1490fa70565444de86b17"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "75aa3ea5157a97f02b38fb87be01812d"
server_key_iv: "1b6dbcfa3ea6931e9f8004582064617b"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.302009 31670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "555596133af1490fa70565444de86b17"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "75aa3ea5157a97f02b38fb87be01812d"
server_key_iv: "1b6dbcfa3ea6931e9f8004582064617b"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.309006 31670 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 09:03:57.313619 31685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.315632 31670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:03:57.315856 31670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "555596133af1490fa70565444de86b17"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "75aa3ea5157a97f02b38fb87be01812d"
server_key_iv: "1b6dbcfa3ea6931e9f8004582064617b"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.316102 31670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:57.338272 31670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:57.343237 31670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:57.343595 31670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:57.344705 31670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:57.346235 31670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:57.346366 31670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.346550 31670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:57.346658 31670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.386211 31670 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:44231
I20260806 09:03:57.386373 31799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:44231 every 8 connection(s)
I20260806 09:03:57.387749 31670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:03:57.397145 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31670
I20260806 09:03:57.397375 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:03:57.398025 28398 external_mini_cluster.cc:1428] Setting key 5f80148f3f50bdda0112d1ad942bab07
Aug 06 09:03:57 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007037, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:57.413774 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:57.390621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:52929 (local address 127.27.187.190:42593)
0806 09:03:57.391159 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:57.391164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:57.392361 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 09:03:57.394727 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:57.394737 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:57.394883 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:57.395027 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:57.397589 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.398340 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.400539 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.400960 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.401109 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:57.405359 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:57.405369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:57.405377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:57.405427 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:57.409050 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:57.410593 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:57.410598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:57.410601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:57.410843 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:57.411483 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:57.411488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:57.411490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:57.411836 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:57.411969 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:57.413215 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0806 09:03:57.413473 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
I20260806 09:03:57.415385 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:57.391237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42593 (local address 127.27.187.129:52929)
0806 09:03:57.392071 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:57.392189 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:57.394217 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:57.395343 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:57.395353 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:57.396312 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:57.397347 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:57.397378 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.398537 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.398543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:57.400360 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:57.400374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.401227 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.401235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:57.401504 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:57.402272 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:57.402298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:57.405049 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:57.409310 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:57.409324 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:57.409351 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:57.410371 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:57.410970 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:57.410985 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:57.410992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:57.411308 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:57.412008 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:57.412019 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:57.412630 (+ 611us) client_negotiation.cc:769] Sending connection context
0806 09:03:57.412986 (+ 356us) client_negotiation.cc:240] Negotiation successful
0806 09:03:57.413661 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":325,"threads_started":1}
I20260806 09:03:57.417755 31800 heartbeater.cc:347] Connected to a master server at 127.27.187.190:42593
I20260806 09:03:57.418119 31800 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:57.419042 31800 heartbeater.cc:511] Master 127.27.187.190:42593 requested a full tablet report, sending...
I20260806 09:03:57.422096 31608 ts_manager.cc:194] Registered new tserver with Master: 555596133af1490fa70565444de86b17 (127.27.187.129:44231)
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:03:57.423853 31608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:52929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:03:57.459333 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:57.826475 31806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:57.826874 31806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:57.826982 31806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:57.834934 31806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:57.835059 31806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:57.835239 31806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:03:57.845700 31806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:57.847085 31806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:57.848515 31806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:03:57.863076 31811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:57.863227 31812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:57.863734 31814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:57.863914 31806 server_base.cc:1034] running on GCE node
I20260806 09:03:57.864600 31806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:57.865653 31806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:57.866853 31806 hybrid_clock.cc:648] HybridClock initialized: now 1786007037866783 us; error 64 us; skew 500 ppm
I20260806 09:03:57.867251 31806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:57 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007037, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:57.871042 31806 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:03:57.873093 31806 webserver.cc:460] Webserver started at http://127.27.187.130:35191/ using document root <none> and password file <none>
I20260806 09:03:57.873962 31806 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:57.874078 31806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:57.874411 31806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:57.877642 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "2f907534066446cd90c8bd0fb9bcf142"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "ba4dc4f831a6694ecdbf7fd5a792169f"
server_key_iv: "46d1a6f46c5fe1695749d26196839e6d"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.878592 31806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "2f907534066446cd90c8bd0fb9bcf142"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "ba4dc4f831a6694ecdbf7fd5a792169f"
server_key_iv: "46d1a6f46c5fe1695749d26196839e6d"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.884739 31806 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 09:03:57.888888 31821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.890740 31806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:03:57.890982 31806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "2f907534066446cd90c8bd0fb9bcf142"
format_stamp: "Formatted at 2026-08-06 09:03:57 on dist-test-slave-8tdl"
server_key: "ba4dc4f831a6694ecdbf7fd5a792169f"
server_key_iv: "46d1a6f46c5fe1695749d26196839e6d"
server_key_version: "encryptionkey@0"
I20260806 09:03:57.891201 31806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:57.920037 31806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:57.925576 31806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:57.925947 31806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:57.927174 31806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:57.928800 31806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:57.928917 31806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.929091 31806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:57.929179 31806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:57.968494 31806 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:37693
I20260806 09:03:57.968503 31936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:37693 every 8 connection(s)
I20260806 09:03:57.970074 31806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:03:57.976202 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31806
I20260806 09:03:57.976397 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:03:57.977005 28398 external_mini_cluster.cc:1428] Setting key 9067eed21b8c4364e79555ff8db83cb5
Aug 06 09:03:57 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007037, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:57.998442 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:57.973299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:35367 (local address 127.27.187.190:42593)
0806 09:03:57.973818 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:57.973823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:57.974995 (+ 1172us) server_negotiation.cc:407] Connection header received
0806 09:03:57.977624 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:57.977631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:57.977788 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:57.977918 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:57.980715 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.981480 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.984476 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.984964 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.985076 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:57.990149 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:57.990155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:57.990159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:57.990200 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:57.993773 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:57.995063 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:57.995068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:57.995071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:57.995233 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:57.995822 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:57.995827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:57.995829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:57.996163 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:57.996350 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:57.997561 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0806 09:03:57.997909 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":232,"threads_started":1}
I20260806 09:03:57.999286 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:57.973939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42593 (local address 127.27.187.130:35367)
0806 09:03:57.974824 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:57.974871 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:57.977234 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:57.978501 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:57.978511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:57.979505 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:57.980468 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:57.980503 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.981672 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.981681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:57.984245 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:57.984269 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:57.985253 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:57.985264 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:57.985690 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:57.986710 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:57.986755 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:57.989815 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:57.994206 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:57.994216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:57.994233 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:57.994887 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:57.995372 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:57.995378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:57.995394 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:57.995641 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:57.996316 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:57.996327 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:57.996741 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 09:03:57.997063 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 09:03:57.997659 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260806 09:03:58.001521 31937 heartbeater.cc:347] Connected to a master server at 127.27.187.190:42593
I20260806 09:03:58.001932 31937 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
I20260806 09:03:58.003387 31937 heartbeater.cc:511] Master 127.27.187.190:42593 requested a full tablet report, sending...
I20260806 09:03:58.005471 31608 ts_manager.cc:194] Registered new tserver with Master: 2f907534066446cd90c8bd0fb9bcf142 (127.27.187.130:37693)
I20260806 09:03:58.006412 31608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:35367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:03:58.044291 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:03:58.399952 31943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:03:58.400472 31943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:03:58.400655 31943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:03:58.408854 31943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:03:58.409031 31943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:03:58.409232 31943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:03:58.420526 31943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:03:58.422122 31943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:03:58.424288 31943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:03:58.428043 31800 heartbeater.cc:503] Master 127.27.187.190:42593 was elected leader, sending a full tablet report...
W20260806 09:03:58.441365 31949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:58.441418 31951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:03:58.441450 31948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:03:58.442880 31943 server_base.cc:1034] running on GCE node
I20260806 09:03:58.443717 31943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:03:58.444977 31943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:03:58.446228 31943 hybrid_clock.cc:648] HybridClock initialized: now 1786007038446179 us; error 43 us; skew 500 ppm
I20260806 09:03:58.446741 31943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:03:58.450487 31943 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:03:58.452437 31943 webserver.cc:460] Webserver started at http://127.27.187.131:36477/ using document root <none> and password file <none>
I20260806 09:03:58.453302 31943 fs_manager.cc:359] Metadata directory not provided
I20260806 09:03:58.453433 31943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:03:58.453727 31943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:03:58.457095 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "e6827fc4dff4476caa297690f6f30aee"
format_stamp: "Formatted at 2026-08-06 09:03:58 on dist-test-slave-8tdl"
server_key: "db57925af2bc552fa4283d690e94d98a"
server_key_iv: "2bd15a15c0d1ea71d834bdf56db25f10"
server_key_version: "encryptionkey@0"
I20260806 09:03:58.458015 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "e6827fc4dff4476caa297690f6f30aee"
format_stamp: "Formatted at 2026-08-06 09:03:58 on dist-test-slave-8tdl"
server_key: "db57925af2bc552fa4283d690e94d98a"
server_key_iv: "2bd15a15c0d1ea71d834bdf56db25f10"
server_key_version: "encryptionkey@0"
I20260806 09:03:58.464527 31943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260806 09:03:58.468896 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:58.470873 31943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:03:58.471098 31943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "e6827fc4dff4476caa297690f6f30aee"
format_stamp: "Formatted at 2026-08-06 09:03:58 on dist-test-slave-8tdl"
server_key: "db57925af2bc552fa4283d690e94d98a"
server_key_iv: "2bd15a15c0d1ea71d834bdf56db25f10"
server_key_version: "encryptionkey@0"
I20260806 09:03:58.471318 31943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:03:58.485509 31943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:03:58.491655 31943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:03:58.492033 31943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:03:58.493191 31943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:03:58.494937 31943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:03:58.495038 31943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:58.495224 31943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:03:58.495324 31943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:03:58.534762 31943 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:46247
I20260806 09:03:58.534798 32075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:46247 every 8 connection(s)
I20260806 09:03:58.536132 31943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:03:58.540455 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 31943
I20260806 09:03:58.540666 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:03:58.541167 28398 external_mini_cluster.cc:1428] Setting key f17db870d8967f058e02174324bef3a0
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:58.560039 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.538854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:48087 (local address 127.27.187.190:42593)
0806 09:03:58.539358 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:58.539363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:58.540512 (+ 1149us) server_negotiation.cc:407] Connection header received
0806 09:03:58.542822 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:58.542828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:58.542942 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:58.543041 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:58.545289 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.546122 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.548355 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.548795 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.548916 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.552819 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:58.552825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:58.552829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:58.552868 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:58.556214 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.557357 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.557361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.557364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.557499 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.558022 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.558026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.558028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.558476 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:58.558614 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:58.559514 (+ 900us) server_negotiation.cc:299] Negotiation successful
0806 09:03:58.559747 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260806 09:03:58.561122 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.539477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42593 (local address 127.27.187.131:48087)
0806 09:03:58.540342 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0806 09:03:58.540381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:03:58.542426 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:03:58.543381 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:03:58.543398 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:03:58.544122 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:03:58.545030 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:58.545062 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.546445 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.546449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:58.548146 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:03:58.548165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.549001 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.549006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:03:58.549251 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.549999 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:03:58.550024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:03:58.552565 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:03:58.556393 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:58.556403 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:58.556425 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:58.557185 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:58.557629 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:03:58.557635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:03:58.557639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:03:58.557866 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:03:58.558614 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:03:58.558620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:03:58.559106 (+ 486us) client_negotiation.cc:769] Sending connection context
0806 09:03:58.559359 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 09:03:58.559805 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":328,"threads_started":1}
I20260806 09:03:58.563124 32076 heartbeater.cc:347] Connected to a master server at 127.27.187.190:42593
I20260806 09:03:58.563517 32076 heartbeater.cc:464] Registering TS with master...
I20260806 09:03:58.564373 32076 heartbeater.cc:511] Master 127.27.187.190:42593 requested a full tablet report, sending...
I20260806 09:03:58.566087 31608 ts_manager.cc:194] Registered new tserver with Master: e6827fc4dff4476caa297690f6f30aee (127.27.187.131:46247)
I20260806 09:03:58.567081 31608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:48087
I20260806 09:03:58.571280 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:03:58.632700 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.615775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.27.187.190:42593)
0806 09:03:58.616076 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:58.616085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:58.616587 (+ 502us) server_negotiation.cc:407] Connection header received
0806 09:03:58.616845 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:58.616849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:58.616937 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:58.617050 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:58.618577 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.619432 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.620928 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.621375 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.621527 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.626360 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:58.626366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:58.626370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:58.626419 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:58.629585 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.630431 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.630442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.630445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.630589 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.631047 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.631053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.631056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.631490 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:58.631772 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:58.632092 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:03:58.632358 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260806 09:03:58.633355 31608 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34082
I20260806 09:03:58.635780 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31670
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:03:58.655934 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.637423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.27.187.129:44231)
0806 09:03:58.638165 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:58.638174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:58.638270 (+ 96us) server_negotiation.cc:407] Connection header received
0806 09:03:58.638343 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:58.638351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:58.638592 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:58.638888 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:58.640534 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.641752 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.643465 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.644142 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.644275 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.649129 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:58.649147 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:58.649165 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:58.649211 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:58.652796 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.653682 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.653690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.653692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.653835 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.654197 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.654200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.654202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.654573 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:58.654889 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:58.655262 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 09:03:58.655510 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":384,"threads_started":1}
W20260806 09:03:58.657456 31795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52716
W20260806 09:03:58.658954 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:58.659181 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31670
I20260806 09:03:58.677862 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31806
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:03:58.697254 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.678954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.27.187.130:37693)
0806 09:03:58.679596 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:58.679604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:58.679632 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:03:58.679707 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:58.679714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:58.679947 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:58.680088 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:58.681440 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.682789 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.684308 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.684950 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.685114 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.689924 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:58.689940 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:58.689957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:58.690009 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:58.693489 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.694385 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.694393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.694396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.694545 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.695169 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.695175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.695179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.695639 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:58.696069 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:58.696431 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 09:03:58.696759 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":340,"threads_started":1}
W20260806 09:03:58.698750 31932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37308
W20260806 09:03:58.700091 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:58.700290 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31806
I20260806 09:03:58.720222 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31943
Aug 06 09:03:58 dist-test-slave-8tdl krb5kdc[31562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:03:58.739847 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:03:58.721473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.27.187.131:46247)
0806 09:03:58.722084 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 09:03:58.722092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:03:58.722133 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:03:58.722198 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:03:58.722204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:03:58.722435 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:03:58.722632 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:03:58.724240 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.725446 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.726980 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:03:58.727740 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:03:58.727895 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:03:58.732873 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:03:58.732889 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:03:58.732904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:03:58.732955 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:03:58.736238 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.737118 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.737132 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.737135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.737409 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:03:58.737941 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:03:58.737944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:03:58.737946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:03:58.738407 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:03:58.738654 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 09:03:58.739121 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 09:03:58.739408 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":310,"threads_started":1}
W20260806 09:03:58.741210 32071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47838
W20260806 09:03:58.742583 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31943: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:03:58.742874 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31943
I20260806 09:03:58.763396 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 31575
I20260806 09:03:58.764919 31639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:03:58.880409 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 31575
2026-08-06T09:03:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4059 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:03:59 dist-test-slave-8tdl krb5kdc[32094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:03:59 dist-test-slave-8tdl krb5kdc[32094](info): set up 2 sockets
Aug 06 09:03:59 dist-test-slave-8tdl krb5kdc[32094](info): commencing operation
krb5kdc: starting...
W20260806 09:04:00.037920 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:00 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:04:00.300768 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45527
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:00.668817 32107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:00.669225 32107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:00.669405 32107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:00.677407 32107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:00.677590 32107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:00.677721 32107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:00.677824 32107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:00.677912 32107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:00.688580 32107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:00.690121 32107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:00.691707 32107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:00.705487 32115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:00.705557 32113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:00.705588 32112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:00.706815 32107 server_base.cc:1034] running on GCE node
I20260806 09:04:00.707754 32107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:00.709609 32107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:00.710865 32107 hybrid_clock.cc:648] HybridClock initialized: now 1786007040710818 us; error 41 us; skew 500 ppm
I20260806 09:04:00.711275 32107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:00 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:00.715001 32107 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:04:00.716899 32107 webserver.cc:460] Webserver started at http://127.27.187.190:38333/ using document root <none> and password file <none>
I20260806 09:04:00.717703 32107 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:00.717872 32107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:00.718286 32107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:00.722043 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "6956a7f699094bce8b6df616cecc5666"
format_stamp: "Formatted at 2026-08-06 09:04:00 on dist-test-slave-8tdl"
server_key: "3ecb90f25686f1da4a6bb14c71783e14"
server_key_iv: "8803f93b8a18b11e0a29550f5d50e4b3"
server_key_version: "encryptionkey@0"
I20260806 09:04:00.723181 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "6956a7f699094bce8b6df616cecc5666"
format_stamp: "Formatted at 2026-08-06 09:04:00 on dist-test-slave-8tdl"
server_key: "3ecb90f25686f1da4a6bb14c71783e14"
server_key_iv: "8803f93b8a18b11e0a29550f5d50e4b3"
server_key_version: "encryptionkey@0"
I20260806 09:04:00.729516 32107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:04:00.733875 32124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:00.735733 32107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:04:00.735989 32107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "6956a7f699094bce8b6df616cecc5666"
format_stamp: "Formatted at 2026-08-06 09:04:00 on dist-test-slave-8tdl"
server_key: "3ecb90f25686f1da4a6bb14c71783e14"
server_key_iv: "8803f93b8a18b11e0a29550f5d50e4b3"
server_key_version: "encryptionkey@0"
I20260806 09:04:00.736245 32107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:00.755692 32107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:00.760742 32107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:00.761089 32107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:00.782218 32107 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45527
I20260806 09:04:00.782250 32176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45527 every 8 connection(s)
I20260806 09:04:00.783840 32107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:00.787441 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32107
I20260806 09:04:00.787708 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:00.788525 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:00.788612 28398 external_mini_cluster.cc:1428] Setting key 14e1bad87cacdbf060419b665b52143e
I20260806 09:04:00.804637 32177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666: Bootstrap starting.
Aug 06 09:04:00 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:00.809091 32177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:00.810701 32177 log.cc:824] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:00.813999 32177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666: No bootstrap required, opened a new log
I20260806 09:04:00.815946 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:00.790966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.27.187.190:45527)
0806 09:04:00.791890 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:00.791911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:00.791964 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:04:00.794335 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:00.794373 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:00.795061 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:00.795603 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:00.797132 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:00.799174 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:00.800698 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:00.801911 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:00.802162 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:00.806805 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:00.806817 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:00.806842 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:00.806904 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:00.811347 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:00.812289 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:00.812297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:00.812304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:00.812495 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:00.812938 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:00.812943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:00.812947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:00.813541 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:00.813861 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:00.814360 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 09:04:00.814867 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":374,"threads_started":1}
I20260806 09:04:00.822902 32177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } }
I20260806 09:04:00.823382 32177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:00.823503 32177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6956a7f699094bce8b6df616cecc5666, State: Initialized, Role: FOLLOWER
I20260806 09:04:00.824158 32177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } }
I20260806 09:04:00.824455 32177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:00.824620 32177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:00.824815 32177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:00.827733 32177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } }
I20260806 09:04:00.828281 32177 leader_election.cc:302] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6956a7f699094bce8b6df616cecc5666; no voters:
I20260806 09:04:00.828879 32177 leader_election.cc:288] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:00.829061 32182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:00.829897 32182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [term 1 LEADER]: Becoming Leader. State: Replica: 6956a7f699094bce8b6df616cecc5666, State: Running, Role: LEADER
I20260806 09:04:00.830464 32182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } }
I20260806 09:04:00.831254 32177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:00.833425 32183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6956a7f699094bce8b6df616cecc5666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } } }
I20260806 09:04:00.833907 32183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:00.833633 32184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6956a7f699094bce8b6df616cecc5666. Latest consensus state: current_term: 1 leader_uuid: "6956a7f699094bce8b6df616cecc5666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956a7f699094bce8b6df616cecc5666" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45527 } } }
I20260806 09:04:00.834157 32184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:00.835007 32187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:00.840847 32187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:00.852334 32187 catalog_manager.cc:1347] Generated new cluster ID: deb0e7896b1140d5869c27c33ceb0902
I20260806 09:04:00.852530 32187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:00.867115 32187 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:00.868672 32187 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:00.883402 32187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6956a7f699094bce8b6df616cecc5666: Generated new TSK 0
I20260806 09:04:00.884640 32187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:04:00.952303 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:01.314476 32203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:01.314893 32203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:01.315055 32203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:01.322777 32203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:01.322959 32203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:01.323156 32203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:01.333720 32203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:01.335372 32203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:01.336838 32203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:01.351920 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:01.351886 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:01.352129 32211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:01.358745 32203 server_base.cc:1034] running on GCE node
I20260806 09:04:01.359597 32203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:01.360852 32203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:01.362129 32203 hybrid_clock.cc:648] HybridClock initialized: now 1786007041362050 us; error 78 us; skew 500 ppm
I20260806 09:04:01.362569 32203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:01 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007041, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:01.366506 32203 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:04:01.368474 32203 webserver.cc:460] Webserver started at http://127.27.187.129:40361/ using document root <none> and password file <none>
I20260806 09:04:01.369315 32203 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:01.369486 32203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:01.369848 32203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:01.373359 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "9020dd70b7d241bd8be4003a2eaed08b"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "f6a7ecd0173db3e7c936b71230f72200"
server_key_iv: "69cd7f7b5663ce2ceca319d32da94a42"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.374356 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "9020dd70b7d241bd8be4003a2eaed08b"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "f6a7ecd0173db3e7c936b71230f72200"
server_key_iv: "69cd7f7b5663ce2ceca319d32da94a42"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.380996 32203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 09:04:01.385234 32218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:01.386884 32203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:01.387085 32203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "9020dd70b7d241bd8be4003a2eaed08b"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "f6a7ecd0173db3e7c936b71230f72200"
server_key_iv: "69cd7f7b5663ce2ceca319d32da94a42"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.387331 32203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:01.412580 32203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:01.418190 32203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:01.418556 32203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:01.419737 32203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:01.421236 32203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:01.421352 32203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:01.421512 32203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:01.421587 32203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:01.459043 32203 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:44671
I20260806 09:04:01.459106 32334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:44671 every 8 connection(s)
I20260806 09:04:01.460554 32203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:01.470230 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32203
I20260806 09:04:01.470505 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:01.471263 28398 external_mini_cluster.cc:1428] Setting key dc8dc6fa3d1799cde31c9d381add082a
Aug 06 09:04:01 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007041, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:01.484687 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:01.463642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:46505 (local address 127.27.187.190:45527)
0806 09:04:01.464269 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:01.464274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:01.465244 (+ 970us) server_negotiation.cc:407] Connection header received
0806 09:04:01.467880 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:01.467887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:01.468004 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:01.468168 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:01.470052 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:01.471029 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:01.473060 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:01.473523 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:01.473688 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:01.477565 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:01.477571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:01.477574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:01.477615 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:01.480586 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:01.481572 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:01.481580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:01.481589 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:01.481736 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:01.482349 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:01.482357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:01.482365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:01.482883 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:01.483259 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:01.484021 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 09:04:01.484301 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260806 09:04:01.485824 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:01.464122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45527 (local address 127.27.187.129:46505)
0806 09:04:01.464993 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:01.465038 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:01.467458 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:01.468474 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:01.468484 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:01.469110 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:01.469820 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:01.469842 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:01.471229 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:01.471233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:01.472898 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:01.472915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:01.473787 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:01.473796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:01.474068 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:01.474883 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:01.474910 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:01.477264 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:01.480854 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:01.480863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:01.480880 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:01.481418 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:01.481862 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:01.481868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:01.481872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:01.482092 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:01.483140 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:01.483149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:01.483598 (+ 449us) client_negotiation.cc:769] Sending connection context
0806 09:04:01.483909 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 09:04:01.484522 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":337,"threads_started":1}
I20260806 09:04:01.487632 32335 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45527
I20260806 09:04:01.488001 32335 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:01.488982 32335 heartbeater.cc:511] Master 127.27.187.190:45527 requested a full tablet report, sending...
I20260806 09:04:01.491875 32141 ts_manager.cc:194] Registered new tserver with Master: 9020dd70b7d241bd8be4003a2eaed08b (127.27.187.129:44671)
I20260806 09:04:01.493522 32141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:46505
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:04:01.531584 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:01.897643 32341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:01.897975 32341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:01.898129 32341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:01.906044 32341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:01.906359 32341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:01.906606 32341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:01.918741 32341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:01.920361 32341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:01.921960 32341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:01.938755 32346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:01.938738 32347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:01.938814 32349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:01.939424 32341 server_base.cc:1034] running on GCE node
I20260806 09:04:01.940171 32341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:01.941659 32341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:01.942953 32341 hybrid_clock.cc:648] HybridClock initialized: now 1786007041942881 us; error 96 us; skew 500 ppm
I20260806 09:04:01.943349 32341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:01 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007041, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:01.947806 32341 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:04:01.949633 32341 webserver.cc:460] Webserver started at http://127.27.187.130:43753/ using document root <none> and password file <none>
I20260806 09:04:01.950424 32341 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:01.950565 32341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:01.950955 32341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:01.954432 32341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "9e418e17766e4ad18730818676c0acce"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "a9657f135d763359b94343ecb64fdd93"
server_key_iv: "bfe4c6f29632093bd1fc55069fc2ddab"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.955457 32341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "9e418e17766e4ad18730818676c0acce"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "a9657f135d763359b94343ecb64fdd93"
server_key_iv: "bfe4c6f29632093bd1fc55069fc2ddab"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.961826 32341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 09:04:01.966387 32356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:01.968465 32341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:04:01.968708 32341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "9e418e17766e4ad18730818676c0acce"
format_stamp: "Formatted at 2026-08-06 09:04:01 on dist-test-slave-8tdl"
server_key: "a9657f135d763359b94343ecb64fdd93"
server_key_iv: "bfe4c6f29632093bd1fc55069fc2ddab"
server_key_version: "encryptionkey@0"
I20260806 09:04:01.968972 32341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:02.011181 32341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:02.017094 32341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:02.017490 32341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:02.018618 32341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:02.020198 32341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:02.020287 32341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:02.020380 32341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:02.020417 32341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:02.058364 32341 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:39213
I20260806 09:04:02.058482 32470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:39213 every 8 connection(s)
I20260806 09:04:02.059828 32341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:02.069375 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32341
I20260806 09:04:02.069692 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:02.070334 28398 external_mini_cluster.cc:1428] Setting key 834f5539775c1973936969c69c65f7b9
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007041, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:02.088845 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.062941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:57187 (local address 127.27.187.190:45527)
0806 09:04:02.063502 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.063507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.064704 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 09:04:02.067208 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.067213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.067300 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.067412 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.069790 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.071008 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.073660 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.074459 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.074621 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.079620 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.079631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.079640 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.079708 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.084139 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.085315 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.085325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.085331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.085533 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.086086 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.086094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.086099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.086581 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.086816 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.087961 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.088304 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":308,"threads_started":1}
I20260806 09:04:02.089775 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.063626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45527 (local address 127.27.187.130:57187)
0806 09:04:02.064509 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:02.064552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:02.066631 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:02.067731 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:02.067740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:02.068605 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:02.069449 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:02.069479 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.071340 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.071348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:02.073405 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:02.073450 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.074820 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.074829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:02.075091 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.075929 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:02.075955 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:02.079018 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:02.084519 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:02.084527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:02.084545 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:02.085122 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:02.085684 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:02.085711 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:02.085714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:02.085946 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:02.086832 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:02.086839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:02.087422 (+ 583us) client_negotiation.cc:769] Sending connection context
0806 09:04:02.087811 (+ 389us) client_negotiation.cc:240] Negotiation successful
0806 09:04:02.088430 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":311,"threads_started":1}
I20260806 09:04:02.091776 32471 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45527
I20260806 09:04:02.092176 32471 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:02.093217 32471 heartbeater.cc:511] Master 127.27.187.190:45527 requested a full tablet report, sending...
I20260806 09:04:02.095816 32141 ts_manager.cc:194] Registered new tserver with Master: 9e418e17766e4ad18730818676c0acce (127.27.187.130:39213)
I20260806 09:04:02.096755 32141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:57187
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:04:02.143287 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:02.497501 32335 heartbeater.cc:503] Master 127.27.187.190:45527 was elected leader, sending a full tablet report...
W20260806 09:04:02.509717 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:02.510049 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:02.510453 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:02.519716 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:02.519865 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:02.520125 32477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:02.531570 32477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:02.533147 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:02.534534 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:02.549301 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:02.550192 32483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:02.550186 32485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:02.551422 32477 server_base.cc:1034] running on GCE node
I20260806 09:04:02.552219 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:02.553355 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:02.554766 32477 hybrid_clock.cc:648] HybridClock initialized: now 1786007042554675 us; error 73 us; skew 500 ppm
I20260806 09:04:02.555131 32477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:02.558861 32477 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:04:02.560592 32477 webserver.cc:460] Webserver started at http://127.27.187.131:45833/ using document root <none> and password file <none>
I20260806 09:04:02.561354 32477 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:02.561471 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:02.561787 32477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:02.565316 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "315798a2c35b4d629af1e6dfc29f7187"
format_stamp: "Formatted at 2026-08-06 09:04:02 on dist-test-slave-8tdl"
server_key: "f69d7597148c5e3779a3ef025a51635c"
server_key_iv: "cdafd4fe3604e85c0faadc1a95c228f1"
server_key_version: "encryptionkey@0"
I20260806 09:04:02.566206 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "315798a2c35b4d629af1e6dfc29f7187"
format_stamp: "Formatted at 2026-08-06 09:04:02 on dist-test-slave-8tdl"
server_key: "f69d7597148c5e3779a3ef025a51635c"
server_key_iv: "cdafd4fe3604e85c0faadc1a95c228f1"
server_key_version: "encryptionkey@0"
I20260806 09:04:02.572654 32477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260806 09:04:02.577433 32492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:02.579418 32477 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:04:02.579629 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "315798a2c35b4d629af1e6dfc29f7187"
format_stamp: "Formatted at 2026-08-06 09:04:02 on dist-test-slave-8tdl"
server_key: "f69d7597148c5e3779a3ef025a51635c"
server_key_iv: "cdafd4fe3604e85c0faadc1a95c228f1"
server_key_version: "encryptionkey@0"
I20260806 09:04:02.579874 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:02.594033 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:02.599195 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:02.599516 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:02.600564 32477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:02.602007 32477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:02.602087 32477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:02.602208 32477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:02.602274 32477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:02.639664 32477 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:35455
I20260806 09:04:02.639739 32606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:35455 every 8 connection(s)
I20260806 09:04:02.641357 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:02.650249 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32477
I20260806 09:04:02.650480 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:02.651130 28398 external_mini_cluster.cc:1428] Setting key dcb75fbd3ea6741d5389c528707b4976
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:02.664860 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.644369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:37235 (local address 127.27.187.190:45527)
0806 09:04:02.644874 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.644879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.645810 (+ 931us) server_negotiation.cc:407] Connection header received
0806 09:04:02.647848 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.647854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.647970 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.648094 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.649812 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.650847 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.652987 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.653622 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.653756 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.657722 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.657734 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.657738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.657778 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.660955 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.661900 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.661907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.661912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.662077 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.662652 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.662661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.662665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.663039 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.663238 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.664076 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.664371 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":293,"threads_started":1}
I20260806 09:04:02.666240 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.644778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45527 (local address 127.27.187.131:37235)
0806 09:04:02.645571 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:02.645615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:02.647508 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:02.648356 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:02.648366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:02.648952 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:02.649614 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:02.649639 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.651020 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.651024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:02.652833 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:02.652853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.653842 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.653848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:02.654065 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.654949 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:02.654974 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:02.657457 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:02.661153 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:02.661165 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:02.661187 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:02.661745 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:02.662219 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:02.662224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:02.662229 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:02.662484 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:02.663203 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:02.663213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:02.663648 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 09:04:02.663970 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 09:04:02.664623 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":306,"threads_started":1}
I20260806 09:04:02.668071 32607 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45527
I20260806 09:04:02.668453 32607 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:02.669238 32607 heartbeater.cc:511] Master 127.27.187.190:45527 requested a full tablet report, sending...
I20260806 09:04:02.671018 32141 ts_manager.cc:194] Registered new tserver with Master: 315798a2c35b4d629af1e6dfc29f7187 (127.27.187.131:35455)
I20260806 09:04:02.671937 32141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:37235
I20260806 09:04:02.680734 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:02.743551 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.723954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.27.187.129:44671)
0806 09:04:02.724740 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.724748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.724777 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:04:02.724845 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.724853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.725086 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.725245 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.726978 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.728276 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.729639 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.730266 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.730421 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.735071 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.735088 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.735104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.735149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.739732 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.740812 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.740821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.740823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.740982 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.741484 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.741490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.741493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.741964 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.742325 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.742704 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.743050 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":458,"threads_started":1}
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:02.765985 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.748915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.27.187.190:45527)
0806 09:04:02.749172 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.749178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.749466 (+ 288us) server_negotiation.cc:407] Connection header received
0806 09:04:02.749670 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.749675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.749775 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.749878 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.751748 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.752578 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.753960 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.754397 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.754511 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.759468 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.759475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.759478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.759525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.762754 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.763623 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.763630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.763634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.763819 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.764285 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.764291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.764300 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.764670 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.764972 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.765351 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.765620 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260806 09:04:02.766785 32141 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060
I20260806 09:04:02.768141 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32203
I20260806 09:04:02.783345 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.769374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.27.187.129:44671)
0806 09:04:02.769674 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.769684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.769993 (+ 309us) server_negotiation.cc:407] Connection header received
0806 09:04:02.770238 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.770242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.770353 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.770471 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.771922 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.772762 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.774142 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.774569 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.774745 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.777306 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.777311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.777314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.777355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.780508 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.781259 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.781264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.781268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.781419 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.781863 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.781873 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.781875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.782150 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.782408 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.782767 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.783003 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:04:02.784197 32330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:02.877228 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32203
I20260806 09:04:02.896154 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32341
Aug 06 09:04:02 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:04:02.914529 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:02.897107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.27.187.130:39213)
0806 09:04:02.897630 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:02.897637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:02.897665 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:04:02.897745 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:02.897752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:02.897981 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:02.898183 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:02.899560 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.900593 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.901935 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:02.902513 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:02.902663 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:02.907346 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:02.907363 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:02.907383 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:02.907424 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:02.911265 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.912171 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.912179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.912181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.912328 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:02.912741 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:02.912747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:02.912750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:02.913124 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:02.913452 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:02.913832 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 09:04:02.914129 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260806 09:04:02.915846 32466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:03.003155 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32341
I20260806 09:04:03.021688 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32477
Aug 06 09:04:03 dist-test-slave-8tdl krb5kdc[32094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:04:03.040201 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:03.022829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.27.187.131:35455)
0806 09:04:03.023398 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:03.023405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:03.023434 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:04:03.023494 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:03.023501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:03.023714 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:03.023884 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:03.025279 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:03.026341 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:03.027843 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:03.028430 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:03.028573 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:03.033396 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:03.033410 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:03.033426 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:03.033471 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:03.036961 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:03.037812 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:03.037820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:03.037822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:03.037965 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:03.038393 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:03.038397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:03.038399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:03.038826 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:03.039173 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:03.039498 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 09:04:03.039803 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":260,"threads_started":1}
I20260806 09:04:03.041771 32601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:03.130148 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32477
I20260806 09:04:03.148473 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32107
I20260806 09:04:03.149865 32172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:03.260295 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32107
2026-08-06T09:04:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4377 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:03 dist-test-slave-8tdl krb5kdc[32627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:03 dist-test-slave-8tdl krb5kdc[32627](info): set up 2 sockets
Aug 06 09:04:03 dist-test-slave-8tdl krb5kdc[32627](info): commencing operation
krb5kdc: starting...
W20260806 09:04:04.414038 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:04 dist-test-slave-8tdl krb5kdc[32627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:04:04.671687 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44951
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:44951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:05.034122 32641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:05.034458 32641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:05.034617 32641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:05.042191 32641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:05.042331 32641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:05.042415 32641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:05.042505 32641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:05.042598 32641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:05.053128 32641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:44951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:05.054570 32641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:05.056033 32641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:05.069306 32646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:05.069271 32647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:05.069358 32649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:05.070024 32641 server_base.cc:1034] running on GCE node
I20260806 09:04:05.071050 32641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:05.072728 32641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:05.073995 32641 hybrid_clock.cc:648] HybridClock initialized: now 1786007045073951 us; error 41 us; skew 500 ppm
I20260806 09:04:05.074393 32641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:05 dist-test-slave-8tdl krb5kdc[32627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:05.083120 32641 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:04:05.084578 32641 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:05.084731 32641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:05.085106 32641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:05.088536 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "b3ec47ab3efb4bea855757912ac85cae"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "7185f68ddbf704d2df844d5153d93654"
server_key_iv: "4c0f328728a9b8803820da3c02903eeb"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.089455 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "b3ec47ab3efb4bea855757912ac85cae"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "7185f68ddbf704d2df844d5153d93654"
server_key_iv: "4c0f328728a9b8803820da3c02903eeb"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.095408 32641 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260806 09:04:05.099509 32657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:05.101006 32641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:05.101315 32641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "b3ec47ab3efb4bea855757912ac85cae"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "7185f68ddbf704d2df844d5153d93654"
server_key_iv: "4c0f328728a9b8803820da3c02903eeb"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.101584 32641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:05.118360 32641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:05.123392 32641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:05.123708 32641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:05.143911 32641 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:44951
I20260806 09:04:05.143929 32710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:44951 every 8 connection(s)
I20260806 09:04:05.144912 32641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:05.148669 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32641
I20260806 09:04:05.148926 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:05.149868 28398 external_mini_cluster.cc:1428] Setting key 5bafdca7f1dd2ef8f5ae677b79f31c7e
I20260806 09:04:05.149911 32711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:05.164176 32711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: Bootstrap starting.
Aug 06 09:04:05 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:05.168094 32711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:05.169425 32711 log.cc:824] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:05.172626 32711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: No bootstrap required, opened a new log
I20260806 09:04:05.175082 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:05.152364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.27.187.190:44951)
0806 09:04:05.153337 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:05.153356 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:05.153422 (+ 66us) server_negotiation.cc:407] Connection header received
0806 09:04:05.155399 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:05.155429 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:05.155948 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:05.156441 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:05.158123 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.159553 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.161046 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.161980 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.162213 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:05.166869 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:05.166878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:05.166894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:05.166940 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:05.170468 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:05.171372 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:05.171380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:05.171387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:05.171572 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:05.172006 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:05.172012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:05.172020 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:05.172586 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:05.172925 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:05.173415 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 09:04:05.173888 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":445,"threads_started":1}
I20260806 09:04:05.181931 32711 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } }
I20260806 09:04:05.182570 32711 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:05.182807 32711 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ec47ab3efb4bea855757912ac85cae, State: Initialized, Role: FOLLOWER
I20260806 09:04:05.183542 32711 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } }
I20260806 09:04:05.183869 32711 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:05.184046 32711 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:05.184242 32711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:05.186852 32711 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } }
I20260806 09:04:05.187495 32711 leader_election.cc:302] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ec47ab3efb4bea855757912ac85cae; no voters:
I20260806 09:04:05.188273 32711 leader_election.cc:288] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:05.188438 32716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:05.189266 32716 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [term 1 LEADER]: Becoming Leader. State: Replica: b3ec47ab3efb4bea855757912ac85cae, State: Running, Role: LEADER
I20260806 09:04:05.189920 32716 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } }
I20260806 09:04:05.191083 32711 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:05.193389 32719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ec47ab3efb4bea855757912ac85cae. Latest consensus state: current_term: 1 leader_uuid: "b3ec47ab3efb4bea855757912ac85cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } } }
I20260806 09:04:05.193780 32719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:05.193539 32717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ec47ab3efb4bea855757912ac85cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ec47ab3efb4bea855757912ac85cae" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44951 } } }
I20260806 09:04:05.194289 32717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:05.195431 32722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:05.202849 32722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:05.214453 32722 catalog_manager.cc:1347] Generated new cluster ID: 51898b1c3d58456182db53d459eca940
I20260806 09:04:05.214624 32722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:05.233028 32722 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:05.234506 32722 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:05.240918 32722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: Generated new TSK 0
I20260806 09:04:05.241920 32722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:04:05.314639 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:05.683666 32739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:05.684021 32739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:05.684147 32739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:05.693362 32739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:05.693516 32739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:05.693658 32739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:05.705770 32739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:05.707435 32739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:05.708900 32739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:05.723896 32745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:05.724049 32744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:05.725927 32747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:05.727739 32739 server_base.cc:1034] running on GCE node
I20260806 09:04:05.728729 32739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:05.729986 32739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:05.731444 32739 hybrid_clock.cc:648] HybridClock initialized: now 1786007045731367 us; error 69 us; skew 500 ppm
I20260806 09:04:05.731904 32739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:05 dist-test-slave-8tdl krb5kdc[32627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:05.735845 32739 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:04:05.737449 32739 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:05.737603 32739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:05.737979 32739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:05.741477 32739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "7e844343a1474f43ab1e5c99f5029b70"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "b26dfa36952c5eb1513b0abddfd69b94"
server_key_iv: "464100c6c77fd5cba27aed22fa7d904b"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.742416 32739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "7e844343a1474f43ab1e5c99f5029b70"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "b26dfa36952c5eb1513b0abddfd69b94"
server_key_iv: "464100c6c77fd5cba27aed22fa7d904b"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.748667 32739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260806 09:04:05.753059 32754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:05.755079 32739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:04:05.755304 32739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "7e844343a1474f43ab1e5c99f5029b70"
format_stamp: "Formatted at 2026-08-06 09:04:05 on dist-test-slave-8tdl"
server_key: "b26dfa36952c5eb1513b0abddfd69b94"
server_key_iv: "464100c6c77fd5cba27aed22fa7d904b"
server_key_version: "encryptionkey@0"
I20260806 09:04:05.755579 32739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:05.777452 32739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:05.783149 32739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:05.783495 32739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:05.784600 32739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:05.786170 32739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:05.786298 32739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:05.786435 32739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:05.786531 32739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:05.825496 32739 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:34067
I20260806 09:04:05.825548 400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:34067 every 8 connection(s)
I20260806 09:04:05.826496 32739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:05.831367 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 32739
I20260806 09:04:05.831624 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:05.832543 28398 external_mini_cluster.cc:1428] Setting key 9847d01cbf06749b7b112097f5fcb1be
Aug 06 09:04:05 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:05.855114 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:05.829609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:41703 (local address 127.27.187.190:44951)
0806 09:04:05.830189 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:05.830196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:05.831347 (+ 1151us) server_negotiation.cc:407] Connection header received
0806 09:04:05.834226 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:05.834232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:05.834309 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:05.834413 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:05.836870 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.837703 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.840574 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.841116 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.841240 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:05.846060 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:05.846071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:05.846080 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:05.846137 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:05.850489 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:05.851705 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:05.851715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:05.851721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:05.851924 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:05.852609 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:05.852618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:05.852624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:05.853095 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:05.853277 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:05.854216 (+ 939us) server_negotiation.cc:299] Negotiation successful
0806 09:04:05.854558 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
I20260806 09:04:05.856077 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:05.830090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44951 (local address 127.27.187.129:41703)
0806 09:04:05.831110 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:05.831176 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:05.833835 (+ 2659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:05.834834 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:05.834844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:05.835647 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:05.836614 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:05.836648 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.837909 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.837918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:05.840325 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:05.840349 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:05.841399 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:05.841410 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:05.841775 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:05.842717 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:05.842757 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:05.845698 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:05.850813 (+ 5115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:05.850823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:05.850841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:05.851508 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:05.852106 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:05.852110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:05.852112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:05.852439 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:05.853309 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:05.853317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:05.853746 (+ 429us) client_negotiation.cc:769] Sending connection context
0806 09:04:05.854017 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 09:04:05.854546 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":390,"threads_started":1}
I20260806 09:04:05.858246 401 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44951
I20260806 09:04:05.858906 401 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:05.859853 401 heartbeater.cc:511] Master 127.27.187.190:44951 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:04:05.862948 32675 ts_manager.cc:194] Registered new tserver with Master: 7e844343a1474f43ab1e5c99f5029b70 (127.27.187.129:34067)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:04:05.865038 32675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:41703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:04:05.899001 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:06.273383 407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:06.273705 407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:06.273872 407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:06.281553 407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:06.281695 407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:06.281872 407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:06.292763 407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:06.294286 407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:06.295747 407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:06.310631 413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:06.310752 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:06.310700 412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:06.312077 407 server_base.cc:1034] running on GCE node
I20260806 09:04:06.312945 407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:06.314049 407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:06.315325 407 hybrid_clock.cc:648] HybridClock initialized: now 1786007046315304 us; error 70 us; skew 500 ppm
I20260806 09:04:06.315761 407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:06 dist-test-slave-8tdl krb5kdc[32627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:06.319988 407 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:04:06.321394 407 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:06.321604 407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:06.322011 407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:06.326306 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "88257ee4802242e99af9ddcca2f45937"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "697dabd00987d32deae15af1a003e7ab"
server_key_iv: "0eb1f19e3277cfb4c255271733bffcb4"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.327488 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "88257ee4802242e99af9ddcca2f45937"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "697dabd00987d32deae15af1a003e7ab"
server_key_iv: "0eb1f19e3277cfb4c255271733bffcb4"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.333248 407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:04:06.337333 421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.339110 407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:06.339315 407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "88257ee4802242e99af9ddcca2f45937"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "697dabd00987d32deae15af1a003e7ab"
server_key_iv: "0eb1f19e3277cfb4c255271733bffcb4"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.339570 407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:06.363824 407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:06.368917 407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:06.369230 407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:06.370311 407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:06.371889 407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:06.371984 407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.372140 407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:06.372219 407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.408490 407 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:45127
I20260806 09:04:06.408608 537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:45127 every 8 connection(s)
I20260806 09:04:06.409433 407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:06.417685 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 407
I20260806 09:04:06.417941 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:06.418512 28398 external_mini_cluster.cc:1428] Setting key 435781fa23adf907c0cb70db8a29cd81
Aug 06 09:04:06 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:06.432571 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:06.412139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:38867 (local address 127.27.187.190:44951)
0806 09:04:06.412634 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:06.412640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:06.413735 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 09:04:06.415728 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:06.415733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:06.415812 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:06.415922 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:06.417580 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.418399 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.420384 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.420918 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.421019 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:06.425596 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:06.425602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:06.425606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:06.425649 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:06.428563 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:06.429469 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:06.429475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:06.429477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:06.429623 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:06.430114 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:06.430118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:06.430120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:06.430469 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:06.430620 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:06.431770 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0806 09:04:06.432112 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":204,"threads_started":1}
I20260806 09:04:06.433881 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:06.412826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44951 (local address 127.27.187.130:38867)
0806 09:04:06.413560 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:06.413599 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:06.415401 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:06.416131 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:06.416139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:06.416745 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:06.417369 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:06.417388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.418577 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.418581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:06.420189 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:06.420213 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.421166 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.421173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:06.421493 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:06.422272 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:06.422309 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:06.425316 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:06.428773 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:06.428784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:06.428808 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:06.429313 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:06.429761 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:06.429766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:06.429770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:06.429978 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:06.430649 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:06.430661 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:06.431206 (+ 545us) client_negotiation.cc:769] Sending connection context
0806 09:04:06.431565 (+ 359us) client_negotiation.cc:240] Negotiation successful
0806 09:04:06.432248 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260806 09:04:06.435614 539 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44951
I20260806 09:04:06.435979 539 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:06.436904 539 heartbeater.cc:511] Master 127.27.187.190:44951 requested a full tablet report, sending...
I20260806 09:04:06.438772 32675 ts_manager.cc:194] Registered new tserver with Master: 88257ee4802242e99af9ddcca2f45937 (127.27.187.130:45127)
I20260806 09:04:06.440120 32675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:38867
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:04:06.480005 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:06.839445 545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:06.839838 545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:06.840018 545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:06.847956 545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:06.848148 545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:06.848325 545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:06.858793 545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:44951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:06.860342 545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:06.861835 545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:04:06.869689 401 heartbeater.cc:503] Master 127.27.187.190:44951 was elected leader, sending a full tablet report...
W20260806 09:04:06.879633 550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:06.879570 551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:06.880272 553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:06.880463 545 server_base.cc:1034] running on GCE node
I20260806 09:04:06.881305 545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:06.882423 545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:06.883648 545 hybrid_clock.cc:648] HybridClock initialized: now 1786007046883526 us; error 114 us; skew 500 ppm
I20260806 09:04:06.884032 545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:06 dist-test-slave-8tdl krb5kdc[32627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:06.887779 545 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:04:06.889267 545 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:06.889427 545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:06.889842 545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:06.893738 545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "f200e4e5db3cfbea84cef81b5e3d84b1"
server_key_iv: "a468a0407ebc12dd646b86c5d793d728"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.894940 545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "f200e4e5db3cfbea84cef81b5e3d84b1"
server_key_iv: "a468a0407ebc12dd646b86c5d793d728"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.901489 545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:04:06.905771 559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.907637 545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:04:06.907840 545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73"
format_stamp: "Formatted at 2026-08-06 09:04:06 on dist-test-slave-8tdl"
server_key: "f200e4e5db3cfbea84cef81b5e3d84b1"
server_key_iv: "a468a0407ebc12dd646b86c5d793d728"
server_key_version: "encryptionkey@0"
I20260806 09:04:06.908097 545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:06.927239 545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:06.932157 545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:06.932693 545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:06.933879 545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:06.935843 545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:06.935981 545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.936125 545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:06.936233 545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:06.975836 545 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33745
I20260806 09:04:06.975906 675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33745 every 8 connection(s)
I20260806 09:04:06.976840 545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:06.986918 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 545
I20260806 09:04:06.987145 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:06.987681 28398 external_mini_cluster.cc:1428] Setting key d82acecff116d1c0aee4d2317417ae9b
Aug 06 09:04:06 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:07.000170 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:06.979780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:47023 (local address 127.27.187.190:44951)
0806 09:04:06.980264 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:06.980269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:06.981169 (+ 900us) server_negotiation.cc:407] Connection header received
0806 09:04:06.983171 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:06.983177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:06.983300 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:06.983421 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:06.985108 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.985943 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.988357 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.988837 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.988991 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:06.993331 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:06.993337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:06.993340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:06.993378 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:06.996420 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:06.997451 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:06.997456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:06.997458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:06.997590 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:06.998070 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:06.998074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:06.998076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:06.998409 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:06.998585 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:06.999459 (+ 874us) server_negotiation.cc:299] Negotiation successful
0806 09:04:06.999750 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
I20260806 09:04:07.001194 679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:06.980217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44951 (local address 127.27.187.131:47023)
0806 09:04:06.980940 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:06.980984 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:06.982825 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:06.983689 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:06.983698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:06.984274 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:06.984898 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:06.984917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.986115 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.986126 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:06.988187 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:06.988202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:06.989157 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:06.989165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:06.989446 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:06.990219 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:06.990246 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:06.993084 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:06.996611 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:06.996622 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:06.996642 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:06.997296 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:06.997721 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:06.997726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:06.997729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:06.997937 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:06.998561 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:06.998570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:06.999024 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 09:04:06.999340 (+ 316us) client_negotiation.cc:240] Negotiation successful
0806 09:04:06.999956 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":307,"threads_started":1}
I20260806 09:04:07.003094 676 heartbeater.cc:347] Connected to a master server at 127.27.187.190:44951
I20260806 09:04:07.003499 676 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:07.004407 676 heartbeater.cc:511] Master 127.27.187.190:44951 requested a full tablet report, sending...
I20260806 09:04:07.006101 32675 ts_manager.cc:194] Registered new tserver with Master: dea70d4e6bc2484c9c1fcc439cc9bb73 (127.27.187.131:33745)
I20260806 09:04:07.007184 32675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:47023
I20260806 09:04:07.017568 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:04:07.036870 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.022663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.27.187.190:44951)
0806 09:04:07.022864 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.022869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.023279 (+ 410us) server_negotiation.cc:407] Connection header received
0806 09:04:07.023420 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.023425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.023499 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.023592 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:07.025131 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.025969 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.027456 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.027905 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.028012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.030558 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:07.030563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:07.030566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:07.030605 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:07.033782 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.034623 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.034628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.034631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.034843 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.035334 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.035339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.035341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.035729 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:07.035950 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.036366 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.036579 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 09:04:07.041792 32675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:07.049023 32675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:07.104984 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.063237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:45127 (local address 127.0.0.1:44484)
0806 09:04:07.089440 (+ 26203us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:07.089480 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:07.089727 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:07.091472 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:07.091485 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:07.091538 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:07.092125 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.092145 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.095640 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.095650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.099255 (+ 3605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.099287 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.103685 (+ 4398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.103699 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.104005 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.104092 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 09:04:07.104276 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 09:04:07.104543 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25976,"thread_start_us":31467,"threads_started":1}
I20260806 09:04:07.106000 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.063960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.27.187.130:45127)
0806 09:04:07.088290 (+ 24330us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.088306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.089959 (+ 1653us) server_negotiation.cc:407] Connection header received
0806 09:04:07.090129 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.090143 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.090560 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.091070 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:07.092702 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.095370 (+ 2668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.099592 (+ 4222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.101786 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.102035 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.102326 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.104416 (+ 2090us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.105549 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24061,"thread_start_us":24365,"threads_started":1}
I20260806 09:04:07.110807 472 tablet_service.cc:1467] Processing CreateTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 (DEFAULT_TABLE table=test-table [id=e6c9604b427c47f79488d417614d6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:07.112308 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.062807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.27.187.131:33745)
0806 09:04:07.102983 (+ 40176us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.102995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.103037 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:04:07.103165 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.103177 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.103494 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.104619 (+ 1125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:07.105719 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.107224 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.109804 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.110945 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.111129 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.111260 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.111686 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.111879 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39954,"thread_start_us":39776,"threads_started":1}
I20260806 09:04:07.112345 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.062493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33745 (local address 127.0.0.1:54010)
0806 09:04:07.064482 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:07.064534 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:07.064702 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:07.103816 (+ 39114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:07.103821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:07.103855 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:07.104382 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.104395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.107375 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.107381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.109572 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.109595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.111177 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.111184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.111407 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.111445 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:04:07.111649 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 09:04:07.111816 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":6,"thread_start_us":1542,"threads_started":1}
I20260806 09:04:07.112622 472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18a14a07d494eb28b44338dc1b6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:07.116138 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.064844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:34067 (local address 127.0.0.1:39904)
0806 09:04:07.103963 (+ 39119us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:07.103984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:07.104155 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:07.105119 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:07.105124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:07.105155 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:07.105572 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.105585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.111334 (+ 5749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.111339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.113729 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.113748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.115249 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.115256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.115423 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.115455 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:04:07.115544 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 09:04:07.115699 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38972,"mutex_wait_us":43,"thread_start_us":38787,"threads_started":1}
I20260806 09:04:07.117285 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.065402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.27.187.129:34067)
0806 09:04:07.103428 (+ 38026us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.103438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.104104 (+ 666us) server_negotiation.cc:407] Connection header received
0806 09:04:07.104354 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.104363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.104612 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.104832 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:07.105797 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.107424 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.113923 (+ 6499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.114970 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.115249 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.116761 (+ 1512us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.116843 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.117029 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37814,"thread_start_us":37650,"threads_started":1}
I20260806 09:04:07.116861 608 tablet_service.cc:1467] Processing CreateTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 (DEFAULT_TABLE table=test-table [id=e6c9604b427c47f79488d417614d6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:07.118474 608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18a14a07d494eb28b44338dc1b6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:07.120919 335 tablet_service.cc:1467] Processing CreateTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 (DEFAULT_TABLE table=test-table [id=e6c9604b427c47f79488d417614d6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:07.122179 335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18a14a07d494eb28b44338dc1b6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:07.131225 695 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Bootstrap starting.
I20260806 09:04:07.134717 696 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Bootstrap starting.
I20260806 09:04:07.135875 695 tablet_bootstrap.cc:654] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:07.136255 697 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Bootstrap starting.
I20260806 09:04:07.137686 695 log.cc:824] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:07.138636 696 tablet_bootstrap.cc:654] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:07.140038 697 tablet_bootstrap.cc:654] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:07.140113 696 log.cc:824] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:07.140712 695 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: No bootstrap required, opened a new log
I20260806 09:04:07.141031 695 ts_tablet_manager.cc:1397] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:04:07.141360 697 log.cc:824] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:07.143539 696 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: No bootstrap required, opened a new log
I20260806 09:04:07.143980 696 ts_tablet_manager.cc:1397] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260806 09:04:07.144762 697 tablet_bootstrap.cc:492] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: No bootstrap required, opened a new log
I20260806 09:04:07.145043 697 ts_tablet_manager.cc:1397] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:04:07.153898 695 raft_consensus.cc:348] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.154443 695 raft_consensus.cc:374] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:07.154536 695 raft_consensus.cc:729] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88257ee4802242e99af9ddcca2f45937, State: Initialized, Role: FOLLOWER
I20260806 09:04:07.155341 695 consensus_queue.cc:260] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.156312 696 raft_consensus.cc:348] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.156309 697 raft_consensus.cc:348] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.156786 697 raft_consensus.cc:374] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:07.156852 696 raft_consensus.cc:374] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:07.156909 697 raft_consensus.cc:729] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e844343a1474f43ab1e5c99f5029b70, State: Initialized, Role: FOLLOWER
I20260806 09:04:07.157079 696 raft_consensus.cc:729] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea70d4e6bc2484c9c1fcc439cc9bb73, State: Initialized, Role: FOLLOWER
I20260806 09:04:07.157644 697 consensus_queue.cc:260] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.157757 696 consensus_queue.cc:260] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.162958 539 heartbeater.cc:503] Master 127.27.187.190:44951 was elected leader, sending a full tablet report...
I20260806 09:04:07.163909 695 ts_tablet_manager.cc:1428] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20260806 09:04:07.166647 696 ts_tablet_manager.cc:1428] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Time spent starting tablet: real 0.022s user 0.012s sys 0.005s
I20260806 09:04:07.167316 676 heartbeater.cc:503] Master 127.27.187.190:44951 was elected leader, sending a full tablet report...
I20260806 09:04:07.168442 697 ts_tablet_manager.cc:1428] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Time spent starting tablet: real 0.023s user 0.014s sys 0.009s
I20260806 09:04:07.179845 701 raft_consensus.cc:482] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:07.180207 701 raft_consensus.cc:504] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.181981 701 leader_election.cc:288] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dea70d4e6bc2484c9c1fcc439cc9bb73 (127.27.187.131:33745), 7e844343a1474f43ab1e5c99f5029b70 (127.27.187.129:34067)
I20260806 09:04:07.188767 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.182468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:33847 (local address 127.27.187.131:33745)
0806 09:04:07.182741 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.182750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.183334 (+ 584us) server_negotiation.cc:407] Connection header received
0806 09:04:07.183585 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.183595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.183698 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.183800 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:07.184400 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.185114 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.187008 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.187812 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.188005 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.188091 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.188378 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.188507 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:04:07.188902 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.182331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33745 (local address 127.27.187.130:33847)
0806 09:04:07.183172 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:07.183212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:07.183395 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:07.183884 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:07.183888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:07.183916 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:07.184249 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.184258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.185284 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.185290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.186848 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.186861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.188043 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.188049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.188209 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.188233 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:04:07.188339 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 09:04:07.188486 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":526,"threads_started":1}
I20260806 09:04:07.189188 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.183576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:57033 (local address 127.27.187.129:34067)
0806 09:04:07.183805 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.183811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.184046 (+ 235us) server_negotiation.cc:407] Connection header received
0806 09:04:07.184274 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.184279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.184349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.184443 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:07.184980 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.185723 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.187480 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.188246 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.188423 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.188501 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.188809 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.188935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:04:07.189873 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.183306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:34067 (local address 127.27.187.130:57033)
0806 09:04:07.183939 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:07.183961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:07.184112 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:07.184456 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:07.184460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:07.184482 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:07.184836 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.184844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.185868 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.185873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.187322 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:07.187333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.188467 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.188474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:07.188660 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.188682 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:04:07.188781 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:04:07.189518 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":1,"thread_start_us":342,"threads_started":1}
I20260806 09:04:07.190240 628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18a14a07d494eb28b44338dc1b6bd85" candidate_uuid: "88257ee4802242e99af9ddcca2f45937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" is_pre_election: true
I20260806 09:04:07.190889 628 raft_consensus.cc:2454] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88257ee4802242e99af9ddcca2f45937 in term 0.
I20260806 09:04:07.190932 356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18a14a07d494eb28b44338dc1b6bd85" candidate_uuid: "88257ee4802242e99af9ddcca2f45937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e844343a1474f43ab1e5c99f5029b70" is_pre_election: true
I20260806 09:04:07.191530 356 raft_consensus.cc:2454] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88257ee4802242e99af9ddcca2f45937 in term 0.
I20260806 09:04:07.191637 425 leader_election.cc:302] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88257ee4802242e99af9ddcca2f45937, dea70d4e6bc2484c9c1fcc439cc9bb73; no voters:
I20260806 09:04:07.192112 701 raft_consensus.cc:2781] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:07.192283 701 raft_consensus.cc:482] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:07.192408 701 raft_consensus.cc:3037] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:07.194792 701 raft_consensus.cc:504] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.195423 701 leader_election.cc:288] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [CANDIDATE]: Term 1 election: Requested vote from peers dea70d4e6bc2484c9c1fcc439cc9bb73 (127.27.187.131:33745), 7e844343a1474f43ab1e5c99f5029b70 (127.27.187.129:34067)
I20260806 09:04:07.196061 628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18a14a07d494eb28b44338dc1b6bd85" candidate_uuid: "88257ee4802242e99af9ddcca2f45937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73"
I20260806 09:04:07.196086 356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18a14a07d494eb28b44338dc1b6bd85" candidate_uuid: "88257ee4802242e99af9ddcca2f45937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e844343a1474f43ab1e5c99f5029b70"
I20260806 09:04:07.196305 628 raft_consensus.cc:3037] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:07.196364 356 raft_consensus.cc:3037] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:07.198779 628 raft_consensus.cc:2454] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88257ee4802242e99af9ddcca2f45937 in term 1.
I20260806 09:04:07.198876 356 raft_consensus.cc:2454] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88257ee4802242e99af9ddcca2f45937 in term 1.
I20260806 09:04:07.199276 425 leader_election.cc:302] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88257ee4802242e99af9ddcca2f45937, dea70d4e6bc2484c9c1fcc439cc9bb73; no voters:
I20260806 09:04:07.199635 701 raft_consensus.cc:2781] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:07.200155 701 raft_consensus.cc:686] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 1 LEADER]: Becoming Leader. State: Replica: 88257ee4802242e99af9ddcca2f45937, State: Running, Role: LEADER
I20260806 09:04:07.200695 701 consensus_queue.cc:237] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } }
I20260806 09:04:07.206282 32674 catalog_manager.cc:5495] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88257ee4802242e99af9ddcca2f45937 (127.27.187.130). New cstate: current_term: 1 leader_uuid: "88257ee4802242e99af9ddcca2f45937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88257ee4802242e99af9ddcca2f45937" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 45127 } health_report { overall_health: HEALTHY } } }
W20260806 09:04:07.228565 677 tablet.cc:2369] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:07.234316 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.228192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.27.187.130:45127)
0806 09:04:07.228421 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.228426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.228515 (+ 89us) server_negotiation.cc:407] Connection header received
0806 09:04:07.228853 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.228858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.228950 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.229080 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:07.229590 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.230353 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.231861 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.232298 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.232400 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.233044 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:07.233618 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:07.233744 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.233889 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.233967 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:04:07.244002 628 raft_consensus.cc:1261] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 1 FOLLOWER]: Refusing update from remote peer 88257ee4802242e99af9ddcca2f45937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:07.244002 356 raft_consensus.cc:1261] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 1 FOLLOWER]: Refusing update from remote peer 88257ee4802242e99af9ddcca2f45937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:07.245266 701 consensus_queue.cc:1035] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea70d4e6bc2484c9c1fcc439cc9bb73" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:07.245666 706 consensus_queue.cc:1035] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e844343a1474f43ab1e5c99f5029b70" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:07.256939 710 mvcc.cc:205] Tried to move back new op lower bound from 7315484865498046464 to 7315484865338462208. Current Snapshot: MvccSnapshot[applied={T|T < 7315484865498046464}]
I20260806 09:04:07.260412 709 mvcc.cc:205] Tried to move back new op lower bound from 7315484865498046464 to 7315484865338462208. Current Snapshot: MvccSnapshot[applied={T|T < 7315484865498046464}]
I20260806 09:04:07.277480 32675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:07.277783 32675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:07.281947 32675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3ec47ab3efb4bea855757912ac85cae: Sending DeleteTablet for 3 replicas of tablet f18a14a07d494eb28b44338dc1b6bd85
I20260806 09:04:07.283442 608 tablet_service.cc:1514] Processing DeleteTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:07 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:54010
I20260806 09:04:07.283461 472 tablet_service.cc:1514] Processing DeleteTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:07 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:44484
I20260806 09:04:07.284578 335 tablet_service.cc:1514] Processing DeleteTablet for tablet f18a14a07d494eb28b44338dc1b6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:07 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:39904
I20260806 09:04:07.284979 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32739
I20260806 09:04:07.295944 718 tablet_replica.cc:330] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: stopping tablet replica
I20260806 09:04:07.298187 718 raft_consensus.cc:2229] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:07.299355 718 raft_consensus.cc:2258] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:07.302031 719 tablet_replica.cc:330] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: stopping tablet replica
I20260806 09:04:07.302597 719 raft_consensus.cc:2229] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:07.302762 720 tablet_replica.cc:330] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: stopping tablet replica
I20260806 09:04:07.303226 719 raft_consensus.cc:2258] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:07.303272 720 raft_consensus.cc:2229] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:07.303836 720 raft_consensus.cc:2258] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:07.306257 718 ts_tablet_manager.cc:1905] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:07.306236 719 ts_tablet_manager.cc:1905] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:07.308085 720 ts_tablet_manager.cc:1905] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:07.314854 718 ts_tablet_manager.cc:1918] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:07.315086 718 log.cc:1196] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/f18a14a07d494eb28b44338dc1b6bd85
I20260806 09:04:07.315727 718 ts_tablet_manager.cc:1939] T f18a14a07d494eb28b44338dc1b6bd85 P 88257ee4802242e99af9ddcca2f45937: Deleting consensus metadata
I20260806 09:04:07.316640 720 ts_tablet_manager.cc:1918] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:07.316845 720 log.cc:1196] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/f18a14a07d494eb28b44338dc1b6bd85
I20260806 09:04:07.317422 720 ts_tablet_manager.cc:1939] T f18a14a07d494eb28b44338dc1b6bd85 P 7e844343a1474f43ab1e5c99f5029b70: Deleting consensus metadata
I20260806 09:04:07.317912 32659 catalog_manager.cc:4841] TS 88257ee4802242e99af9ddcca2f45937 (127.27.187.130:45127): tablet f18a14a07d494eb28b44338dc1b6bd85 (table test-table [id=e6c9604b427c47f79488d417614d6489]) successfully deleted
Aug 06 09:04:07 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:07.319368 32661 catalog_manager.cc:4841] TS 7e844343a1474f43ab1e5c99f5029b70 (127.27.187.129:34067): tablet f18a14a07d494eb28b44338dc1b6bd85 (table test-table [id=e6c9604b427c47f79488d417614d6489]) successfully deleted
I20260806 09:04:07.321794 719 ts_tablet_manager.cc:1918] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:07.322012 719 log.cc:1196] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/f18a14a07d494eb28b44338dc1b6bd85
I20260806 09:04:07.322605 719 ts_tablet_manager.cc:1939] T f18a14a07d494eb28b44338dc1b6bd85 P dea70d4e6bc2484c9c1fcc439cc9bb73: Deleting consensus metadata
I20260806 09:04:07.324772 32661 catalog_manager.cc:4841] TS dea70d4e6bc2484c9c1fcc439cc9bb73 (127.27.187.131:33745): tablet f18a14a07d494eb28b44338dc1b6bd85 (table test-table [id=e6c9604b427c47f79488d417614d6489]) successfully deleted
I20260806 09:04:07.327978 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.286526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.27.187.129:34067)
0806 09:04:07.287340 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.287350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.301158 (+ 13808us) server_negotiation.cc:407] Connection header received
0806 09:04:07.301836 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.301842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.301941 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.302031 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:07.304742 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.306040 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.308754 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.309534 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.309658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.320634 (+ 10976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:07.320650 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:07.320671 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:07.320719 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:07.324813 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.325667 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.325675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.325677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.325836 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.326262 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.326266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.326268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.326651 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:07.327058 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.327363 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.327601 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260806 09:04:07.328706 396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:07.428339 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32739
I20260806 09:04:07.456305 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 407
Aug 06 09:04:07 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:04:07.478538 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.457639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.27.187.130:45127)
0806 09:04:07.457872 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.457882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.457905 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:04:07.457985 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.457989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.458060 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.458206 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:07.459652 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.460572 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.461886 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.462611 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.463001 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.468826 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:07.468845 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:07.468866 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:07.468912 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:07.475225 (+ 6313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.476136 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.476151 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.476153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.476329 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.476782 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.476788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.476796 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.477191 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:07.477618 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.477872 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.478136 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:04:07.479460 533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:07.601673 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 407
I20260806 09:04:07.623274 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 545
Aug 06 09:04:07 dist-test-slave-8tdl krb5kdc[32627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:04:07.640817 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:07.624042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.27.187.131:33745)
0806 09:04:07.624268 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:07.624272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:07.624291 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:04:07.624411 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:07.624415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:07.624480 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:07.624576 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:07.625941 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.626867 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.628329 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:07.628760 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:07.628878 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:07.634238 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:07.634258 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:07.634275 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:07.634321 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:07.637460 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.638332 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.638340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.638342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.638507 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:07.639133 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:07.639137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:07.639139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:07.639551 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:07.639815 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:07.640278 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 09:04:07.640531 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:04:07.641523 670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:07.739553 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 545
I20260806 09:04:07.760074 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 32641
I20260806 09:04:07.761317 32706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:07.875427 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 32641
2026-08-06T09:04:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4614 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T09:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:07Z Disabled control of system clock
I20260806 09:04:07.956032 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38143
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:04:08.309657 729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:08.309988 729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:08.310128 729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:08.317636 729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:08.317770 729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:08.317811 729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:08.317863 729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:08.317909 729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:08.327524 729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:08.328855 729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:08.330785 729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:08.343513 734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:08.343817 735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:08.343838 737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:08.344611 729 server_base.cc:1034] running on GCE node
I20260806 09:04:08.345435 729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:08.346940 729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:08.348151 729 hybrid_clock.cc:648] HybridClock initialized: now 1786007048348105 us; error 34 us; skew 500 ppm
I20260806 09:04:08.348594 729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:08.351357 729 webserver.cc:460] Webserver started at http://127.27.187.190:39253/ using document root <none> and password file <none>
I20260806 09:04:08.352129 729 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:08.352281 729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:08.352641 729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:08.356312 729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "f6d889da067f488d8c4632bf44021206"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "23c7748db932866391c3740364806a3e"
server_key_iv: "0e93396e4345d71676eb155f6f03d4cc"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.357266 729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "f6d889da067f488d8c4632bf44021206"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "23c7748db932866391c3740364806a3e"
server_key_iv: "0e93396e4345d71676eb155f6f03d4cc"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.363595 729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:04:08.367647 743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:08.369557 729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:08.369776 729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "f6d889da067f488d8c4632bf44021206"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "23c7748db932866391c3740364806a3e"
server_key_iv: "0e93396e4345d71676eb155f6f03d4cc"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.370038 729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:08.376438 729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:08.377269 729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:08.377547 729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:08.397431 729 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38143
I20260806 09:04:08.397470 794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38143 every 8 connection(s)
I20260806 09:04:08.399055 729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:08.401394 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 729
I20260806 09:04:08.401623 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:08.402318 28398 external_mini_cluster.cc:1428] Setting key 09ed5ea79318ac49bbe95e294eaa4014
I20260806 09:04:08.403863 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:08.413393 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:08.404711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.27.187.190:38143)
0806 09:04:08.405970 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:08.405988 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:08.406049 (+ 61us) server_negotiation.cc:407] Connection header received
0806 09:04:08.408032 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:08.408067 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:08.408845 (+ 778us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:08.409498 (+ 653us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:08.410859 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:08.410871 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:08.410895 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:08.410916 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:08.411044 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:08.411071 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:08.411416 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:08.411617 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 09:04:08.411951 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":389,"threads_started":1}
I20260806 09:04:08.420096 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: Bootstrap starting.
I20260806 09:04:08.423669 795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:08.424722 795 log.cc:824] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:08.427807 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: No bootstrap required, opened a new log
I20260806 09:04:08.435456 795 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } }
I20260806 09:04:08.435880 795 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:08.435984 795 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d889da067f488d8c4632bf44021206, State: Initialized, Role: FOLLOWER
I20260806 09:04:08.436607 795 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } }
I20260806 09:04:08.436894 795 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:08.437052 795 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:08.437249 795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:08.440423 795 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } }
I20260806 09:04:08.441085 795 leader_election.cc:302] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d889da067f488d8c4632bf44021206; no voters:
I20260806 09:04:08.441869 795 leader_election.cc:288] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:08.442011 800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:08.442780 800 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [term 1 LEADER]: Becoming Leader. State: Replica: f6d889da067f488d8c4632bf44021206, State: Running, Role: LEADER
I20260806 09:04:08.443452 800 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } }
I20260806 09:04:08.444196 795 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:08.447463 801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d889da067f488d8c4632bf44021206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } } }
I20260806 09:04:08.447463 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d889da067f488d8c4632bf44021206. Latest consensus state: current_term: 1 leader_uuid: "f6d889da067f488d8c4632bf44021206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d889da067f488d8c4632bf44021206" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38143 } } }
I20260806 09:04:08.447924 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:08.447922 801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:08.449663 805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:08.455546 805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:08.465899 805 catalog_manager.cc:1347] Generated new cluster ID: b0494ab5b53e4d4f90e2da840ae1a535
I20260806 09:04:08.466058 805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:08.479833 805 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:08.480433 805 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:08.491428 805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: Generated new TSK 0
I20260806 09:04:08.492468 805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:04:08.504719 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--builtin_ntp_servers=127.27.187.148:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:04:08.898584 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:08.898980 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:08.899127 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:08.906873 821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:08.906996 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:08.907159 821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:08.917621 821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:08.919234 821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:08.921300 821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:08.936360 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:08.936391 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:08.936467 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:08.937356 821 server_base.cc:1034] running on GCE node
I20260806 09:04:08.938071 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:08.939313 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:08.940568 821 hybrid_clock.cc:648] HybridClock initialized: now 1786007048940498 us; error 70 us; skew 500 ppm
I20260806 09:04:08.940934 821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:08.943738 821 webserver.cc:460] Webserver started at http://127.27.187.129:45935/ using document root <none> and password file <none>
I20260806 09:04:08.944593 821 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:08.944739 821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:08.945093 821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:08.948527 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "edd5b9d198ac4d4f805e6fe836fd7524"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "121b32f5fe7ed32329fef8a0909daccf"
server_key_iv: "ae307a5f6803ded960d4e84bdc85aff4"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.949532 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "edd5b9d198ac4d4f805e6fe836fd7524"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "121b32f5fe7ed32329fef8a0909daccf"
server_key_iv: "ae307a5f6803ded960d4e84bdc85aff4"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.956004 821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:04:08.960325 835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:08.961961 821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:04:08.962162 821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "edd5b9d198ac4d4f805e6fe836fd7524"
format_stamp: "Formatted at 2026-08-06 09:04:08 on dist-test-slave-8tdl"
server_key: "121b32f5fe7ed32329fef8a0909daccf"
server_key_iv: "ae307a5f6803ded960d4e84bdc85aff4"
server_key_version: "encryptionkey@0"
I20260806 09:04:08.962412 821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:08.968876 821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:08.969831 821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:08.970129 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:08.971407 821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:08.972977 821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:08.973097 821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:08.973230 821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:08.973315 821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.012403 821 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:33725
I20260806 09:04:09.012467 948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:33725 every 8 connection(s)
I20260806 09:04:09.013989 821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:09.015182 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 821
I20260806 09:04:09.015398 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:09.016260 28398 external_mini_cluster.cc:1428] Setting key 383118dfd454f90903d4d28abab786e5
I20260806 09:04:09.019731 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--builtin_ntp_servers=127.27.187.148:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:04:09.024554 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.017413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:44459 (local address 127.27.187.190:38143)
0806 09:04:09.017910 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:09.017915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:09.019483 (+ 1568us) server_negotiation.cc:407] Connection header received
0806 09:04:09.021617 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:09.021623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:09.021700 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:09.021840 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:09.023587 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:09.023594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:09.023600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:09.023624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:09.023681 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:09.023736 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:09.023899 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:09.024110 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 09:04:09.024194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
I20260806 09:04:09.025630 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.018172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38143 (local address 127.27.187.129:44459)
0806 09:04:09.019322 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:09.019362 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:09.021295 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:09.022109 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:09.022119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:09.022228 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:09.023144 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:09.023147 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:09.023264 (+ 117us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:09.023373 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:09.023849 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:09.023873 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:09.023882 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:04:09.024222 (+ 340us) client_negotiation.cc:240] Negotiation successful
0806 09:04:09.024545 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":336,"threads_started":1}
I20260806 09:04:09.027420 949 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38143
I20260806 09:04:09.027969 949 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:09.028828 949 heartbeater.cc:511] Master 127.27.187.190:38143 requested a full tablet report, sending...
I20260806 09:04:09.031539 760 ts_manager.cc:194] Registered new tserver with Master: edd5b9d198ac4d4f805e6fe836fd7524 (127.27.187.129:33725)
W20260806 09:04:09.378189 953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:09.378554 953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:09.378754 953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:09.386796 953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:09.386952 953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:09.387136 953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:09.397116 953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:09.398432 953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:09.400278 953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:09.414088 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:09.414104 960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:09.414626 962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:09.414852 953 server_base.cc:1034] running on GCE node
I20260806 09:04:09.415542 953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:09.416635 953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:09.417882 953 hybrid_clock.cc:648] HybridClock initialized: now 1786007049417829 us; error 54 us; skew 500 ppm
I20260806 09:04:09.418294 953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:09.421000 953 webserver.cc:460] Webserver started at http://127.27.187.130:41605/ using document root <none> and password file <none>
I20260806 09:04:09.421788 953 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:09.421927 953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:09.422278 953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:09.425390 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "75024d4d68a34d70bc08e67c45cb44be"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "15be037b0ecd32e8cb6bb46b8724f661"
server_key_iv: "ce1696e8cdaa2ce753396a3f1f34f602"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.426343 953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "75024d4d68a34d70bc08e67c45cb44be"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "15be037b0ecd32e8cb6bb46b8724f661"
server_key_iv: "ce1696e8cdaa2ce753396a3f1f34f602"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.432260 953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:04:09.436549 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.438294 953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:04:09.438514 953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "75024d4d68a34d70bc08e67c45cb44be"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "15be037b0ecd32e8cb6bb46b8724f661"
server_key_iv: "ce1696e8cdaa2ce753396a3f1f34f602"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.438838 953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:09.447074 953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:09.448089 953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:09.448397 953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:09.449508 953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:09.451102 953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:09.451238 953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.451372 953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:09.451460 953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.488132 953 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:43825
I20260806 09:04:09.488204 1081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:43825 every 8 connection(s)
I20260806 09:04:09.489573 953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:09.498274 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 953
I20260806 09:04:09.498473 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:09.498764 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.492535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:33663 (local address 127.27.187.190:38143)
0806 09:04:09.492757 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:09.492764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:09.494280 (+ 1516us) server_negotiation.cc:407] Connection header received
0806 09:04:09.496298 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:09.496310 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:09.496372 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:09.496467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:09.497884 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:09.497889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:09.497892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:09.497911 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:09.497946 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:09.497958 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:09.498077 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:09.498349 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 09:04:09.498449 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:04:09.499078 28398 external_mini_cluster.cc:1428] Setting key 3f94295124e718c2e1419e41ad0edc4b
I20260806 09:04:09.499977 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.493109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38143 (local address 127.27.187.130:33663)
0806 09:04:09.494112 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:09.494149 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:09.495990 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:09.496676 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:09.496684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:09.496779 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:09.497545 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:09.497548 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:09.497587 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:09.497682 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:09.498065 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:09.498077 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:09.498086 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:04:09.498414 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 09:04:09.498733 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
I20260806 09:04:09.501567 1082 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38143
I20260806 09:04:09.501936 1082 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:09.502545 1082 heartbeater.cc:511] Master 127.27.187.190:38143 requested a full tablet report, sending...
I20260806 09:04:09.502568 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--builtin_ntp_servers=127.27.187.148:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:04:09.504141 760 ts_manager.cc:194] Registered new tserver with Master: 75024d4d68a34d70bc08e67c45cb44be (127.27.187.130:43825)
W20260806 09:04:09.861390 1085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:09.861706 1085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:09.861800 1085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:09.869634 1085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:09.869756 1085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:09.869868 1085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:09.879506 1085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:09.880802 1085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:09.882666 1085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:09.896735 1090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:09.896790 1093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:09.896584 1091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:09.898031 1085 server_base.cc:1034] running on GCE node
I20260806 09:04:09.898847 1085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:09.899866 1085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:09.901078 1085 hybrid_clock.cc:648] HybridClock initialized: now 1786007049900990 us; error 86 us; skew 500 ppm
I20260806 09:04:09.901473 1085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:09.903925 1085 webserver.cc:460] Webserver started at http://127.27.187.131:37287/ using document root <none> and password file <none>
I20260806 09:04:09.904650 1085 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:09.904742 1085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:09.905115 1085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:09.908314 1085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "046c01a003f44f3380897687848c0f15"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "705c1fc3fb63dfda472578c187b1b943"
server_key_iv: "58a54530acac34237c7d434c43ec240a"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.909264 1085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "046c01a003f44f3380897687848c0f15"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "705c1fc3fb63dfda472578c187b1b943"
server_key_iv: "58a54530acac34237c7d434c43ec240a"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.915185 1085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:04:09.919214 1099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.920724 1085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:04:09.920914 1085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "046c01a003f44f3380897687848c0f15"
format_stamp: "Formatted at 2026-08-06 09:04:09 on dist-test-slave-8tdl"
server_key: "705c1fc3fb63dfda472578c187b1b943"
server_key_iv: "58a54530acac34237c7d434c43ec240a"
server_key_version: "encryptionkey@0"
I20260806 09:04:09.921154 1085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:09.927485 1085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:09.928386 1085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:09.928684 1085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:09.929682 1085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:09.931229 1085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:09.931368 1085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.931514 1085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:09.931603 1085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:09.967926 1085 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:35065
I20260806 09:04:09.968139 1214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:35065 every 8 connection(s)
I20260806 09:04:09.969317 1085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:09.978605 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.972022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49987 (local address 127.27.187.190:38143)
0806 09:04:09.972299 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:09.972304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:09.973891 (+ 1587us) server_negotiation.cc:407] Connection header received
0806 09:04:09.976015 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:09.976020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:09.976080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:09.976187 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:09.977776 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:09.977780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:09.977784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:09.977802 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:09.977835 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:09.977847 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:09.977949 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:09.978307 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 09:04:09.978369 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260806 09:04:09.978580 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1085
I20260806 09:04:09.978796 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:09.979380 28398 external_mini_cluster.cc:1428] Setting key 5a7635e9d149f5f06d0f52ebad9b9369
I20260806 09:04:09.980022 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:09.972663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38143 (local address 127.27.187.131:49987)
0806 09:04:09.973719 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:09.973767 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:09.975720 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:09.976581 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:09.976589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:09.976695 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:09.977439 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:09.977442 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:09.977484 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:09.977572 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:09.977936 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:09.977943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:09.977949 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:04:09.978373 (+ 424us) client_negotiation.cc:240] Negotiation successful
0806 09:04:09.978671 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":315,"threads_started":1}
I20260806 09:04:09.981516 1215 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38143
I20260806 09:04:09.981846 1215 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:09.982481 1215 heartbeater.cc:511] Master 127.27.187.190:38143 requested a full tablet report, sending...
I20260806 09:04:09.983943 760 ts_manager.cc:194] Registered new tserver with Master: 046c01a003f44f3380897687848c0f15 (127.27.187.131:35065)
I20260806 09:04:09.995466 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:04:10.003469 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.000528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.27.187.190:38143)
0806 09:04:10.000795 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.000806 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.001286 (+ 480us) server_negotiation.cc:407] Connection header received
0806 09:04:10.001370 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.001375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.001436 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.001532 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.002547 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.002553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.002556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.002580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.002626 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.002640 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.002839 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.003062 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.003126 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:04:10.008559 760 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:10.016726 760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:10.035487 949 heartbeater.cc:503] Master 127.27.187.190:38143 was elected leader, sending a full tablet report...
I20260806 09:04:10.047082 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.030321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52448)
0806 09:04:10.031318 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.031369 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:10.031534 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:10.044077 (+ 12543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:10.044085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:10.044207 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:10.045199 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:10.045203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:10.045234 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:10.045333 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:10.045835 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:10.045849 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:10.045858 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:04:10.046035 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 09:04:10.046136 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":486,"threads_started":1}
I20260806 09:04:10.047701 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.030587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.27.187.129:33725)
0806 09:04:10.043335 (+ 12748us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.043349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.043396 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:04:10.043501 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.043513 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.043834 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.044052 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.045557 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.045568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.045584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.045603 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.045669 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.045691 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.047124 (+ 1433us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.047195 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.047327 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12508,"thread_start_us":12356,"threads_started":1}
I20260806 09:04:10.049477 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.032026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.27.187.131:35065)
0806 09:04:10.042905 (+ 10879us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.042920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.044263 (+ 1343us) server_negotiation.cc:407] Connection header received
0806 09:04:10.044416 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.044428 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.044745 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.044902 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.048333 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.048344 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.048359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.048390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.048455 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.048474 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.048650 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.048972 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.049079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10612,"thread_start_us":10377,"threads_started":1}
I20260806 09:04:10.049701 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.031319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:35065 (local address 127.0.0.1:33178)
0806 09:04:10.043866 (+ 12547us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.043893 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:10.044047 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:10.046246 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:10.046264 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:10.046293 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:10.048068 (+ 1775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:10.048072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:10.048091 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:10.048138 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:10.048604 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:10.048612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:10.048618 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:04:10.048808 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 09:04:10.048865 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12357,"thread_start_us":12115,"threads_started":1}
I20260806 09:04:10.050954 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.031901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:43825 (local address 127.0.0.1:52478)
0806 09:04:10.047139 (+ 15238us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.047162 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:10.047273 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:10.048819 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:10.048823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:10.048847 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:10.049601 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:10.049605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:10.049622 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:10.049697 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:10.050228 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:10.050232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:10.050236 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 09:04:10.050372 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:04:10.050431 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15082,"thread_start_us":14936,"threads_started":1}
I20260806 09:04:10.051645 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.032556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.27.187.130:43825)
0806 09:04:10.046650 (+ 14094us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.046700 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.047411 (+ 711us) server_negotiation.cc:407] Connection header received
0806 09:04:10.047487 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.047495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.047747 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.047900 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.049898 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.049909 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.049924 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.049954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.050015 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.050037 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.050171 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.051236 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.051339 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13905,"thread_start_us":14127,"threads_started":1}
I20260806 09:04:10.052122 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7727f2eb244a2491f9410def6246a0 (DEFAULT_TABLE table=test-table [id=b76be1716c0f4f2d9880c1e0cc674167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:10.053767 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7727f2eb244a2491f9410def6246a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:10.053714 1149 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7727f2eb244a2491f9410def6246a0 (DEFAULT_TABLE table=test-table [id=b76be1716c0f4f2d9880c1e0cc674167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:10.055253 1149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7727f2eb244a2491f9410def6246a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:10.055056 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7727f2eb244a2491f9410def6246a0 (DEFAULT_TABLE table=test-table [id=b76be1716c0f4f2d9880c1e0cc674167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:10.056365 1016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7727f2eb244a2491f9410def6246a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:10.068562 1234 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Bootstrap starting.
I20260806 09:04:10.073081 1234 tablet_bootstrap.cc:654] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:10.073266 1236 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Bootstrap starting.
I20260806 09:04:10.074411 1234 log.cc:824] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:10.079913 1236 tablet_bootstrap.cc:654] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:10.080900 1235 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Bootstrap starting.
I20260806 09:04:10.081718 1236 log.cc:824] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:10.084878 1235 tablet_bootstrap.cc:654] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:10.084992 1234 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: No bootstrap required, opened a new log
I20260806 09:04:10.085286 1234 ts_tablet_manager.cc:1397] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20260806 09:04:10.086280 1235 log.cc:824] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:10.087841 1236 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: No bootstrap required, opened a new log
I20260806 09:04:10.088224 1236 ts_tablet_manager.cc:1397] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.002s
I20260806 09:04:10.089762 1235 tablet_bootstrap.cc:492] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: No bootstrap required, opened a new log
I20260806 09:04:10.090081 1235 ts_tablet_manager.cc:1397] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 09:04:10.094151 1234 raft_consensus.cc:348] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.094584 1234 raft_consensus.cc:374] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:10.094748 1234 raft_consensus.cc:729] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75024d4d68a34d70bc08e67c45cb44be, State: Initialized, Role: FOLLOWER
I20260806 09:04:10.095458 1234 consensus_queue.cc:260] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.096581 1082 heartbeater.cc:503] Master 127.27.187.190:38143 was elected leader, sending a full tablet report...
I20260806 09:04:10.098294 1234 ts_tablet_manager.cc:1428] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260806 09:04:10.099517 1236 raft_consensus.cc:348] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.100069 1236 raft_consensus.cc:374] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:10.100174 1236 raft_consensus.cc:729] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046c01a003f44f3380897687848c0f15, State: Initialized, Role: FOLLOWER
I20260806 09:04:10.100957 1236 consensus_queue.cc:260] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.102072 1235 raft_consensus.cc:348] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.102607 1235 raft_consensus.cc:374] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:10.102730 1235 raft_consensus.cc:729] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd5b9d198ac4d4f805e6fe836fd7524, State: Initialized, Role: FOLLOWER
I20260806 09:04:10.103606 1236 ts_tablet_manager.cc:1428] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260806 09:04:10.103389 1235 consensus_queue.cc:260] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.103873 1215 heartbeater.cc:503] Master 127.27.187.190:38143 was elected leader, sending a full tablet report...
I20260806 09:04:10.105298 1235 ts_tablet_manager.cc:1428] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260806 09:04:10.202096 1240 raft_consensus.cc:482] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:10.202530 1240 raft_consensus.cc:504] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.204816 1240 leader_election.cc:288] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd5b9d198ac4d4f805e6fe836fd7524 (127.27.187.129:33725), 046c01a003f44f3380897687848c0f15 (127.27.187.131:35065)
I20260806 09:04:10.216001 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.205303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:37889 (local address 127.27.187.131:35065)
0806 09:04:10.205532 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.205539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.206433 (+ 894us) server_negotiation.cc:407] Connection header received
0806 09:04:10.206554 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.206559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.206618 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.206819 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.215180 (+ 8361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.215188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.215193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.215222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.215274 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.215293 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.215466 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.215668 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.215731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:04:10.216001 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.205411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:56707 (local address 127.27.187.129:33725)
0806 09:04:10.205601 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.205610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.206371 (+ 761us) server_negotiation.cc:407] Connection header received
0806 09:04:10.206544 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.206550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.206618 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.206774 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.215180 (+ 8406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.215187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.215193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.215222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.215273 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.215293 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.215466 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.215632 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.215710 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:04:10.216477 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.205124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.27.187.130:56707)
0806 09:04:10.206207 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.206229 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:10.206393 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:10.206802 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:10.206807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:10.206836 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:10.214814 (+ 7978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:10.214819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:10.214847 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:10.214926 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:10.215469 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:10.215475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:10.215482 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:04:10.215778 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 09:04:10.215871 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":27,"thread_start_us":545,"threads_started":1}
I20260806 09:04:10.216500 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.205315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:35065 (local address 127.27.187.130:37889)
0806 09:04:10.206313 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.206336 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:10.206428 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:10.206822 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:10.206826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:10.206842 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:10.214814 (+ 7972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:04:10.214819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:10.214847 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:04:10.214926 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:10.215469 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:10.215475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:10.215482 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:04:10.215764 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 09:04:10.215853 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":497,"threads_started":1}
I20260806 09:04:10.217813 1170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7727f2eb244a2491f9410def6246a0" candidate_uuid: "75024d4d68a34d70bc08e67c45cb44be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "046c01a003f44f3380897687848c0f15" is_pre_election: true
I20260806 09:04:10.217813 903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7727f2eb244a2491f9410def6246a0" candidate_uuid: "75024d4d68a34d70bc08e67c45cb44be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" is_pre_election: true
I20260806 09:04:10.218586 1170 raft_consensus.cc:2454] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75024d4d68a34d70bc08e67c45cb44be in term 0.
I20260806 09:04:10.218586 903 raft_consensus.cc:2454] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75024d4d68a34d70bc08e67c45cb44be in term 0.
I20260806 09:04:10.219585 971 leader_election.cc:302] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75024d4d68a34d70bc08e67c45cb44be, edd5b9d198ac4d4f805e6fe836fd7524; no voters:
I20260806 09:04:10.220266 1240 raft_consensus.cc:2781] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:10.220439 1240 raft_consensus.cc:482] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:10.220578 1240 raft_consensus.cc:3037] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:04:10.220979 1216 tablet.cc:2369] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:10.223354 1240 raft_consensus.cc:504] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.224083 1240 leader_election.cc:288] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [CANDIDATE]: Term 1 election: Requested vote from peers edd5b9d198ac4d4f805e6fe836fd7524 (127.27.187.129:33725), 046c01a003f44f3380897687848c0f15 (127.27.187.131:35065)
I20260806 09:04:10.224781 1170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7727f2eb244a2491f9410def6246a0" candidate_uuid: "75024d4d68a34d70bc08e67c45cb44be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "046c01a003f44f3380897687848c0f15"
I20260806 09:04:10.224793 903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7727f2eb244a2491f9410def6246a0" candidate_uuid: "75024d4d68a34d70bc08e67c45cb44be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd5b9d198ac4d4f805e6fe836fd7524"
I20260806 09:04:10.225039 1170 raft_consensus.cc:3037] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:10.225044 903 raft_consensus.cc:3037] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:10.227592 1170 raft_consensus.cc:2454] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75024d4d68a34d70bc08e67c45cb44be in term 1.
I20260806 09:04:10.227758 903 raft_consensus.cc:2454] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75024d4d68a34d70bc08e67c45cb44be in term 1.
I20260806 09:04:10.228286 972 leader_election.cc:302] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 046c01a003f44f3380897687848c0f15, 75024d4d68a34d70bc08e67c45cb44be; no voters:
I20260806 09:04:10.228686 1240 raft_consensus.cc:2781] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:10.229215 1240 raft_consensus.cc:686] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 1 LEADER]: Becoming Leader. State: Replica: 75024d4d68a34d70bc08e67c45cb44be, State: Running, Role: LEADER
I20260806 09:04:10.229728 1240 consensus_queue.cc:237] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } }
I20260806 09:04:10.235327 760 catalog_manager.cc:5495] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be reported cstate change: term changed from 0 to 1, leader changed from <none> to 75024d4d68a34d70bc08e67c45cb44be (127.27.187.130). New cstate: current_term: 1 leader_uuid: "75024d4d68a34d70bc08e67c45cb44be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75024d4d68a34d70bc08e67c45cb44be" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 } health_report { overall_health: UNKNOWN } } }
W20260806 09:04:10.241889 1083 tablet.cc:2369] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:10.252075 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.249525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.27.187.130:43825)
0806 09:04:10.249786 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.249795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.249820 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:04:10.249959 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.249964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.250054 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.250187 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.251331 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.251339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.251344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.251370 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.251409 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.251423 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.251579 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.251665 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.251737 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:04:10.262082 1170 raft_consensus.cc:1261] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 1 FOLLOWER]: Refusing update from remote peer 75024d4d68a34d70bc08e67c45cb44be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:10.262099 903 raft_consensus.cc:1261] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 1 FOLLOWER]: Refusing update from remote peer 75024d4d68a34d70bc08e67c45cb44be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:10.263233 1245 consensus_queue.cc:1035] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd5b9d198ac4d4f805e6fe836fd7524" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:10.263648 1240 consensus_queue.cc:1035] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [LEADER]: Connected to new peer: Peer: permanent_uuid: "046c01a003f44f3380897687848c0f15" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:04:10.266907 950 tablet.cc:2369] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:10.280696 1251 mvcc.cc:205] Tried to move back new op lower bound from 7315484877861339136 to 7315484877745405952. Current Snapshot: MvccSnapshot[applied={T|T < 7315484877861339136}]
I20260806 09:04:10.293130 759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:10.293506 759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:10.298054 759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6d889da067f488d8c4632bf44021206: Sending DeleteTablet for 3 replicas of tablet 3c7727f2eb244a2491f9410def6246a0
I20260806 09:04:10.299876 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7727f2eb244a2491f9410def6246a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:10 UTC) from {username='slave'} at 127.0.0.1:52478
I20260806 09:04:10.299867 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7727f2eb244a2491f9410def6246a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:10 UTC) from {username='slave'} at 127.0.0.1:52448
I20260806 09:04:10.299875 1149 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7727f2eb244a2491f9410def6246a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:10 UTC) from {username='slave'} at 127.0.0.1:33178
I20260806 09:04:10.302424 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 821
I20260806 09:04:10.312728 1260 tablet_replica.cc:330] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: stopping tablet replica
I20260806 09:04:10.314734 1260 raft_consensus.cc:2229] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:10.316732 1260 raft_consensus.cc:2258] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:10.318974 1259 tablet_replica.cc:330] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: stopping tablet replica
I20260806 09:04:10.319275 1261 tablet_replica.cc:330] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: stopping tablet replica
I20260806 09:04:10.319821 1261 raft_consensus.cc:2229] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:10.320447 1261 raft_consensus.cc:2258] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:10.321300 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.303917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.27.187.129:33725)
0806 09:04:10.304152 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.304162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.317986 (+ 13824us) server_negotiation.cc:407] Connection header received
0806 09:04:10.318602 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.318610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.318780 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.318879 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.320328 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.320336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.320340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.320365 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.320427 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.320452 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.320567 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.320899 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.320993 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:04:10.321713 1260 ts_tablet_manager.cc:1905] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:10.322074 944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:10.322368 1259 raft_consensus.cc:2229] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:10.323155 1259 raft_consensus.cc:2258] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:10.327029 1259 ts_tablet_manager.cc:1905] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:10.332875 1259 ts_tablet_manager.cc:1918] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:10.333204 1259 log.cc:1196] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/3c7727f2eb244a2491f9410def6246a0
I20260806 09:04:10.333886 1259 ts_tablet_manager.cc:1939] T 3c7727f2eb244a2491f9410def6246a0 P 046c01a003f44f3380897687848c0f15: Deleting consensus metadata
I20260806 09:04:10.336275 747 catalog_manager.cc:4841] TS 046c01a003f44f3380897687848c0f15 (127.27.187.131:35065): tablet 3c7727f2eb244a2491f9410def6246a0 (table test-table [id=b76be1716c0f4f2d9880c1e0cc674167]) successfully deleted
I20260806 09:04:10.337976 1260 ts_tablet_manager.cc:1918] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:10.338408 1260 log.cc:1196] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/3c7727f2eb244a2491f9410def6246a0
I20260806 09:04:10.339097 1260 ts_tablet_manager.cc:1939] T 3c7727f2eb244a2491f9410def6246a0 P 75024d4d68a34d70bc08e67c45cb44be: Deleting consensus metadata
I20260806 09:04:10.340890 746 catalog_manager.cc:4841] TS 75024d4d68a34d70bc08e67c45cb44be (127.27.187.130:43825): tablet 3c7727f2eb244a2491f9410def6246a0 (table test-table [id=b76be1716c0f4f2d9880c1e0cc674167]) successfully deleted
I20260806 09:04:10.442893 1261 ts_tablet_manager.cc:1905] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:10.449653 1261 ts_tablet_manager.cc:1918] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:10.449911 1261 log.cc:1196] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/3c7727f2eb244a2491f9410def6246a0
I20260806 09:04:10.450441 1261 ts_tablet_manager.cc:1939] T 3c7727f2eb244a2491f9410def6246a0 P edd5b9d198ac4d4f805e6fe836fd7524: Deleting consensus metadata
I20260806 09:04:10.451150 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 821
W20260806 09:04:10.470095 746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:33725 (error 108)
I20260806 09:04:10.470363 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 953
I20260806 09:04:10.472364 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.471818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52466)
0806 09:04:10.472032 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.472200 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 09:04:10.472774 746 catalog_manager.cc:4584] TS edd5b9d198ac4d4f805e6fe836fd7524 (127.27.187.129:33725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c7727f2eb244a2491f9410def6246a0: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
I20260806 09:04:10.474509 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.471318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.27.187.130:43825)
0806 09:04:10.471504 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.471510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.471545 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:04:10.471808 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.471813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.471916 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.472019 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.473574 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.473581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.473586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.473610 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.473652 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.473678 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.473808 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.473991 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.474056 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:04:10.475800 1077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:10.527122 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.526496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52482)
0806 09:04:10.526783 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.526901 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 09:04:10.528337 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.527903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52484)
0806 09:04:10.528077 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.528184 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:04:10.588665 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 953
I20260806 09:04:10.591622 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.591043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52494)
0806 09:04:10.591331 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.591472 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260806 09:04:10.592896 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.592370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52496)
0806 09:04:10.592596 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.592718 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 09:04:10.610353 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1085
I20260806 09:04:10.613776 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.611252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.27.187.131:35065)
0806 09:04:10.611477 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:10.611482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:10.611739 (+ 257us) server_negotiation.cc:407] Connection header received
0806 09:04:10.611892 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:10.611897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:10.611992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:10.612092 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:10.613158 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:10.613163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:10.613167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:10.613185 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:10.613224 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:10.613238 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:10.613379 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:10.613419 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 09:04:10.613472 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:04:10.614488 1210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:10.673614 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.672926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52506)
0806 09:04:10.673237 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.673386 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260806 09:04:10.675033 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:10.674502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33725 (local address 127.0.0.1:52522)
0806 09:04:10.674768 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:10.674865 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 09:04:10.707564 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1085
I20260806 09:04:10.727047 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 729
I20260806 09:04:10.728528 790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:10.843218 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 729
2026-08-06T09:04:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2966 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 09:04:10.888003 28398 mini_oidc.cc:152] Starting JWKS server
I20260806 09:04:10.891070 28398 webserver.cc:460] Webserver started at https://127.0.0.1:35097/ using document root <none> and password file <none>
I20260806 09:04:10.891455 28398 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:04:10.892165 28398 webserver.cc:460] Webserver started at http://127.0.0.1:41313/ using document root <none> and password file <none>
2026-08-06T09:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:10Z Disabled control of system clock
I20260806 09:04:10.928962 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45343
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/testchainca.cert with env {}
W20260806 09:04:11.273872 1273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:11.274219 1273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:11.274369 1273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:11.282097 1273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:11.282240 1273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:11.282337 1273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:11.282428 1273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:11.282529 1273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:04:11.282622 1273 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:04:11.282713 1273 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35097/jwks.json
I20260806 09:04:11.294001 1273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:11.295452 1273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:11.296902 1273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:11.310817 1278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:11.310853 1279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:11.310904 1281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:11.312358 1273 server_base.cc:1034] running on GCE node
I20260806 09:04:11.313356 1273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:11.315032 1273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:11.316327 1273 hybrid_clock.cc:648] HybridClock initialized: now 1786007051316254 us; error 59 us; skew 500 ppm
I20260806 09:04:11.316741 1273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:11.319381 1273 webserver.cc:460] Webserver started at http://127.27.187.190:37475/ using document root <none> and password file <none>
I20260806 09:04:11.320130 1273 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:11.320271 1273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:11.320604 1273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:11.323954 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd"
format_stamp: "Formatted at 2026-08-06 09:04:11 on dist-test-slave-8tdl"
server_key: "9414922977199b2e28ab4fab2b9983c8"
server_key_iv: "96143340e17de62a6f9fe088358b145e"
server_key_version: "encryptionkey@0"
I20260806 09:04:11.324873 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd"
format_stamp: "Formatted at 2026-08-06 09:04:11 on dist-test-slave-8tdl"
server_key: "9414922977199b2e28ab4fab2b9983c8"
server_key_iv: "96143340e17de62a6f9fe088358b145e"
server_key_version: "encryptionkey@0"
I20260806 09:04:11.330753 1273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:04:11.334556 1287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:11.336153 1273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:11.336392 1273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd"
format_stamp: "Formatted at 2026-08-06 09:04:11 on dist-test-slave-8tdl"
server_key: "9414922977199b2e28ab4fab2b9983c8"
server_key_iv: "96143340e17de62a6f9fe088358b145e"
server_key_version: "encryptionkey@0"
I20260806 09:04:11.336633 1273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:11.373672 1273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:11.374667 1273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:11.375042 1273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:11.394958 1273 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45343
I20260806 09:04:11.394984 1339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45343 every 8 connection(s)
I20260806 09:04:11.396520 1273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:11.400785 1340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:11.404978 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1273
I20260806 09:04:11.405254 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:11.405954 28398 external_mini_cluster.cc:1428] Setting key be3eb8035d33b1040281658101b3a9e2
I20260806 09:04:11.413242 1340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd: Bootstrap starting.
I20260806 09:04:11.417326 1340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:11.418800 1340 log.cc:824] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:11.422272 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:11.408120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.27.187.190:45343)
0806 09:04:11.409221 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:11.409236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:11.409284 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:04:11.411680 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:11.411710 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:11.412384 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:11.412784 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:11.413553 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.415099 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.417230 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.418108 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.418278 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:11.419720 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:11.419736 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:11.419778 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:11.420559 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:11.420649 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:11.420677 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:11.420779 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:11.421004 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 09:04:11.421291 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":390,"threads_started":1}
I20260806 09:04:11.422370 1340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd: No bootstrap required, opened a new log
I20260806 09:04:11.426548 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:04:11.431005 1340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } }
I20260806 09:04:11.431416 1340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:11.431524 1340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1b72c7c2564e2c80dbd6b8e002acdd, State: Initialized, Role: FOLLOWER
I20260806 09:04:11.432169 1340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } }
I20260806 09:04:11.432415 1340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:11.432556 1340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:11.432726 1340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:11.435211 1340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } }
I20260806 09:04:11.435765 1340 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1b72c7c2564e2c80dbd6b8e002acdd; no voters:
I20260806 09:04:11.436405 1340 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:11.436538 1345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:11.437302 1345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [term 1 LEADER]: Becoming Leader. State: Replica: 6d1b72c7c2564e2c80dbd6b8e002acdd, State: Running, Role: LEADER
I20260806 09:04:11.437853 1345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } }
I20260806 09:04:11.438643 1340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:11.441321 1347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1b72c7c2564e2c80dbd6b8e002acdd. Latest consensus state: current_term: 1 leader_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } } }
I20260806 09:04:11.441730 1347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:11.442605 1346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b72c7c2564e2c80dbd6b8e002acdd" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45343 } } }
I20260806 09:04:11.443075 1346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:11.444095 1350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:11.449941 1350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:11.460140 1350 catalog_manager.cc:1347] Generated new cluster ID: 6697dd650fe5429bb14fef668b7304db
I20260806 09:04:11.460301 1350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:11.501469 1350 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:11.503110 1350 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:11.511489 1350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1b72c7c2564e2c80dbd6b8e002acdd: Generated new TSK 0
I20260806 09:04:11.512492 1350 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:04:11.565606 1370 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 09:04:11.565999 1370 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:04:11.567063 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:11.542595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.27.187.190:45343)
0806 09:04:11.542857 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:11.542866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:11.543607 (+ 741us) server_negotiation.cc:407] Connection header received
0806 09:04:11.544116 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:11.544122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:11.544342 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:11.544490 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:04:11.545240 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.546093 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.548495 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.548980 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.549086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:11.549883 (+ 797us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:04:11.566414 (+ 16531us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:04:11.566818 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 09:04:11.567235 1343 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 09:04:11.569445 1369 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:11.542558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45343 (local address 127.0.0.1:38416)
0806 09:04:11.543455 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:11.543477 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:11.543718 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:11.544560 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:11.544566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:11.544580 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:04:11.545013 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:11.545025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.546269 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.546276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:11.548261 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:11.548279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:11.549182 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:11.549188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:11.549391 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:11.549485 (+ 94us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:04:11.566807 (+ 17322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:04:11.567252 (+ 445us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.187.190:45343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":450,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.187.190:45343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 09:04:11.571193 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1273
I20260806 09:04:11.572476 1335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:11.673417 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1273
2026-08-06T09:04:11Z chronyd exiting
I20260806 09:04:11.729094 28398 test_util.cc:175] -----------------------------------------------
I20260806 09:04:11.729308 28398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007003447047-28398-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (846 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 09:04:11.733418 28398 mini_oidc.cc:152] Starting JWKS server
I20260806 09:04:11.735448 28398 webserver.cc:460] Webserver started at https://127.0.0.1:40683/ using document root <none> and password file <none>
I20260806 09:04:11.735707 28398 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:04:11.736483 28398 webserver.cc:460] Webserver started at http://127.0.0.1:41569/ using document root <none> and password file <none>
2026-08-06T09:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:11Z Disabled control of system clock
I20260806 09:04:11.774224 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:46295
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:46295
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40683/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/testchainca.cert with env {}
W20260806 09:04:12.141672 1377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:12.142082 1377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:12.142251 1377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:12.149994 1377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:12.150184 1377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:12.150296 1377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:12.150410 1377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:12.150489 1377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:04:12.150585 1377 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:04:12.150662 1377 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40683/jwks.json
I20260806 09:04:12.161501 1377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:46295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40683/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:12.163134 1377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:12.164666 1377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:12.178764 1385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:12.179044 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:12.178797 1387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:12.179885 1377 server_base.cc:1034] running on GCE node
I20260806 09:04:12.180860 1377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:12.182503 1377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:12.183720 1377 hybrid_clock.cc:648] HybridClock initialized: now 1786007052183672 us; error 41 us; skew 500 ppm
I20260806 09:04:12.184171 1377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:12.186910 1377 webserver.cc:460] Webserver started at http://127.27.187.190:38609/ using document root <none> and password file <none>
I20260806 09:04:12.187721 1377 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:12.187826 1377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:12.188174 1377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:12.191479 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "ae170a7382984dea87f2e4d5b86eb504"
format_stamp: "Formatted at 2026-08-06 09:04:12 on dist-test-slave-8tdl"
server_key: "8eedcefc2486a0bbd214a81a8763a302"
server_key_iv: "aeeb23b8fdb41f5624fb20136d1f3183"
server_key_version: "encryptionkey@0"
I20260806 09:04:12.192375 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "ae170a7382984dea87f2e4d5b86eb504"
format_stamp: "Formatted at 2026-08-06 09:04:12 on dist-test-slave-8tdl"
server_key: "8eedcefc2486a0bbd214a81a8763a302"
server_key_iv: "aeeb23b8fdb41f5624fb20136d1f3183"
server_key_version: "encryptionkey@0"
I20260806 09:04:12.198449 1377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:04:12.202502 1393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:12.204341 1377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:04:12.204577 1377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "ae170a7382984dea87f2e4d5b86eb504"
format_stamp: "Formatted at 2026-08-06 09:04:12 on dist-test-slave-8tdl"
server_key: "8eedcefc2486a0bbd214a81a8763a302"
server_key_iv: "aeeb23b8fdb41f5624fb20136d1f3183"
server_key_version: "encryptionkey@0"
I20260806 09:04:12.204839 1377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:12.228696 1377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:12.229660 1377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:12.229993 1377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:12.251133 1377 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:46295
I20260806 09:04:12.251152 1446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:46295 every 8 connection(s)
I20260806 09:04:12.252635 1377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:12.256820 1447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:12.261489 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1377
I20260806 09:04:12.261725 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:12.262609 28398 external_mini_cluster.cc:1428] Setting key a4c7e4d60eac8a91f83e8230ad498928
I20260806 09:04:12.269600 1447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504: Bootstrap starting.
I20260806 09:04:12.273195 1447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:12.274487 1447 log.cc:824] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:12.278424 1447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504: No bootstrap required, opened a new log
I20260806 09:04:12.278810 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:12.264947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.27.187.190:46295)
0806 09:04:12.266193 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:12.266207 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:12.266257 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:04:12.268353 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:12.268381 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:12.268945 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:12.269417 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:12.270063 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.271617 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.273172 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.274161 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.274412 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:12.275858 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:12.275873 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:12.275899 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:12.276579 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:12.276665 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:12.276688 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:12.276785 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:12.277163 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 09:04:12.277561 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":439,"threads_started":1}
I20260806 09:04:12.283175 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:04:12.287551 1447 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } }
I20260806 09:04:12.287968 1447 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:12.288079 1447 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae170a7382984dea87f2e4d5b86eb504, State: Initialized, Role: FOLLOWER
I20260806 09:04:12.288661 1447 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } }
I20260806 09:04:12.288934 1447 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:12.289088 1447 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:12.289273 1447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:12.291603 1447 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } }
I20260806 09:04:12.292164 1447 leader_election.cc:302] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae170a7382984dea87f2e4d5b86eb504; no voters:
I20260806 09:04:12.292896 1447 leader_election.cc:288] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:12.293045 1452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:12.294093 1452 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [term 1 LEADER]: Becoming Leader. State: Replica: ae170a7382984dea87f2e4d5b86eb504, State: Running, Role: LEADER
I20260806 09:04:12.294749 1452 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } }
I20260806 09:04:12.295243 1447 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:12.298059 1454 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae170a7382984dea87f2e4d5b86eb504. Latest consensus state: current_term: 1 leader_uuid: "ae170a7382984dea87f2e4d5b86eb504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } } }
I20260806 09:04:12.298202 1453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae170a7382984dea87f2e4d5b86eb504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae170a7382984dea87f2e4d5b86eb504" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46295 } } }
I20260806 09:04:12.298749 1454 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:12.298794 1453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:12.302994 1460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:12.309219 1460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:12.320640 1460 catalog_manager.cc:1347] Generated new cluster ID: 3de4840f3f01496da93583668fb1bf49
I20260806 09:04:12.320854 1460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:12.336602 1460 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:12.338143 1460 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:12.352491 1460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae170a7382984dea87f2e4d5b86eb504: Generated new TSK 0
I20260806 09:04:12.353492 1460 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:04:12.395716 1478 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 09:04:12.395970 1478 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:04:12.397006 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:12.367421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.27.187.190:46295)
0806 09:04:12.367645 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:12.367653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:12.368059 (+ 406us) server_negotiation.cc:407] Connection header received
0806 09:04:12.368344 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:12.368348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:12.368429 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:12.368597 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:04:12.369094 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.369845 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.371323 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.371819 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.371974 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:12.372477 (+ 503us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:04:12.396356 (+ 23879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40683/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:04:12.396769 (+ 413us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40683/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 09:04:12.397157 1450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40683/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 09:04:12.397425 1476 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:12.367308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:46295 (local address 127.0.0.1:55436)
0806 09:04:12.367943 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:12.367960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:12.368135 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:12.368613 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:12.368618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:12.368630 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:04:12.368934 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:12.368943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.369996 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.370001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:12.371164 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:12.371176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:12.372003 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:12.372008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:12.372175 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:12.372212 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:04:12.396663 (+ 24451us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40683/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:04:12.396971 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.187.190:46295: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40683/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":368,"threads_started":1}
I20260806 09:04:12.398587 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1377
I20260806 09:04:12.400204 1442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:12.499235 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1377
2026-08-06T09:04:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (824 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 09:04:12.558601 28398 mini_oidc.cc:152] Starting JWKS server
I20260806 09:04:12.560680 28398 webserver.cc:460] Webserver started at https://127.0.0.1:33977/ using document root <none> and password file <none>
I20260806 09:04:12.560969 28398 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:04:12.561764 28398 webserver.cc:460] Webserver started at http://127.0.0.1:38383/ using document root <none> and password file <none>
2026-08-06T09:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:12Z Disabled control of system clock
I20260806 09:04:12.602371 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44631
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:44631
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:04:12.961369 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:12.961774 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:12.961932 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:12.969597 1485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:12.969790 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:12.969892 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:12.970001 1485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:12.970093 1485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:04:12.970186 1485 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:04:12.970266 1485 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33977/jwks.json
I20260806 09:04:12.980232 1485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:44631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:12.981681 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:12.983654 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:12.997099 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:12.997069 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:12.996987 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:12.998142 1485 server_base.cc:1034] running on GCE node
I20260806 09:04:12.999183 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:13.000931 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:13.002274 1485 hybrid_clock.cc:648] HybridClock initialized: now 1786007053002227 us; error 38 us; skew 500 ppm
I20260806 09:04:13.002800 1485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:13.005590 1485 webserver.cc:460] Webserver started at http://127.27.187.190:34279/ using document root <none> and password file <none>
I20260806 09:04:13.006429 1485 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:13.006533 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:13.007164 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:13.010835 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "6195e62d4f5c4fb3ba645e52cc087cb6"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "e9c0a0225cb28899d30b6aa42d426ee8"
server_key_iv: "8c3e168c10b9b3e071f24add3e2b3057"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.011868 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "6195e62d4f5c4fb3ba645e52cc087cb6"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "e9c0a0225cb28899d30b6aa42d426ee8"
server_key_iv: "8c3e168c10b9b3e071f24add3e2b3057"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.017778 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 09:04:13.021881 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:13.023609 1485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:13.023836 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "6195e62d4f5c4fb3ba645e52cc087cb6"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "e9c0a0225cb28899d30b6aa42d426ee8"
server_key_iv: "8c3e168c10b9b3e071f24add3e2b3057"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.024094 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:13.045528 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:13.046597 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:13.046984 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:13.068075 1485 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:44631
I20260806 09:04:13.068120 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:44631 every 8 connection(s)
I20260806 09:04:13.069620 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:13.073944 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:13.079351 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1485
I20260806 09:04:13.079622 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:13.080485 28398 external_mini_cluster.cc:1428] Setting key c3ea8a087698a2b3f921408e076844c2
I20260806 09:04:13.086948 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6: Bootstrap starting.
I20260806 09:04:13.090356 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:13.091667 1553 log.cc:824] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:13.095641 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6: No bootstrap required, opened a new log
I20260806 09:04:13.096426 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:13.082767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.27.187.190:44631)
0806 09:04:13.084011 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:13.084026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:13.084085 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:04:13.085944 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:13.085975 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:13.086517 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:13.087025 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:13.087709 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.089080 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.090524 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.091446 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.091695 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:13.093142 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:13.093152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:13.093170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:13.093948 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:13.094073 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:13.094106 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:13.094235 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:13.094614 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 09:04:13.095022 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":422,"threads_started":1}
I20260806 09:04:13.100971 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:04:13.105458 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } }
I20260806 09:04:13.105891 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:13.105969 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6195e62d4f5c4fb3ba645e52cc087cb6, State: Initialized, Role: FOLLOWER
I20260806 09:04:13.106501 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } }
I20260806 09:04:13.106834 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:13.106953 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:13.107084 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:13.109746 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } }
I20260806 09:04:13.110391 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6195e62d4f5c4fb3ba645e52cc087cb6; no voters:
I20260806 09:04:13.111114 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:13.111333 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:13.112128 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [term 1 LEADER]: Becoming Leader. State: Replica: 6195e62d4f5c4fb3ba645e52cc087cb6, State: Running, Role: LEADER
I20260806 09:04:13.112684 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } }
I20260806 09:04:13.113441 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:13.116058 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } } }
I20260806 09:04:13.116493 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:13.116168 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6195e62d4f5c4fb3ba645e52cc087cb6. Latest consensus state: current_term: 1 leader_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6195e62d4f5c4fb3ba645e52cc087cb6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 44631 } } }
I20260806 09:04:13.116631 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:13.123459 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:13.129107 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:13.139158 1565 catalog_manager.cc:1347] Generated new cluster ID: 8bebef2ce3134a88a1b61fc2fd760861
I20260806 09:04:13.139319 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:13.171610 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:13.173580 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:13.183476 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6195e62d4f5c4fb3ba645e52cc087cb6: Generated new TSK 0
I20260806 09:04:13.184636 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:04:13.219009 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:13.209658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.27.187.190:44631)
0806 09:04:13.209829 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:13.209834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:13.210253 (+ 419us) server_negotiation.cc:407] Connection header received
0806 09:04:13.210527 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:13.210534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:13.210653 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:13.210865 (+ 212us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:04:13.211385 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.212567 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.214104 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.214897 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.215043 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:13.215735 (+ 692us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:04:13.218271 (+ 2536us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:04:13.218642 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 09:04:13.219167 1557 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 09:04:13.220822 1583 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:13.209481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:44631 (local address 127.0.0.1:53860)
0806 09:04:13.210135 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:13.210152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:13.210306 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:13.210910 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:13.210914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:13.210925 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:04:13.211213 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:13.211221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.212747 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.212752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:13.213959 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:13.213972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.215224 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.215230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:13.215382 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:13.215422 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:04:13.218535 (+ 3113us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:04:13.220378 (+ 1843us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.187.190:44631: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":1479,"thread_start_us":360,"threads_started":1}
I20260806 09:04:13.222198 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1485
I20260806 09:04:13.223630 1548 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 09:04:13.231174 1584 webserver.cc:533] Webserver: sslize failed:
E20260806 09:04:13.231318 1584 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:04:13.321434 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1485
2026-08-06T09:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (821 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T09:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:13Z Disabled control of system clock
I20260806 09:04:13.416631 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40679
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:40679
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/testchainca.cert with env {}
W20260806 09:04:13.777163 1589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:13.777506 1589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:13.777669 1589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:13.786254 1589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:13.786412 1589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:13.786494 1589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:13.786599 1589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:13.786669 1589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:04:13.786809 1589 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:04:13.786872 1589 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 09:04:13.797821 1589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:40679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:13.799434 1589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:13.800985 1589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:13.814615 1595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:13.814584 1597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:13.814724 1594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:13.815692 1589 server_base.cc:1034] running on GCE node
I20260806 09:04:13.816769 1589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:13.818527 1589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:13.819820 1589 hybrid_clock.cc:648] HybridClock initialized: now 1786007053819771 us; error 91 us; skew 500 ppm
I20260806 09:04:13.820247 1589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:04:13.826999 1589 webserver.cc:460] Webserver started at http://127.27.187.190:38375/ using document root <none> and password file <none>
I20260806 09:04:13.827872 1589 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:13.828032 1589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:13.828372 1589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:13.831733 1589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "c5daeafed76546eba8006d5601ce4d0b"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "65019c57ab976dab61679e64df7cd92d"
server_key_iv: "8ccc4794da41a27d46524f5f18046c41"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.832618 1589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "c5daeafed76546eba8006d5601ce4d0b"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "65019c57ab976dab61679e64df7cd92d"
server_key_iv: "8ccc4794da41a27d46524f5f18046c41"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.838429 1589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:04:13.842367 1603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:13.844061 1589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:13.844285 1589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "c5daeafed76546eba8006d5601ce4d0b"
format_stamp: "Formatted at 2026-08-06 09:04:13 on dist-test-slave-8tdl"
server_key: "65019c57ab976dab61679e64df7cd92d"
server_key_iv: "8ccc4794da41a27d46524f5f18046c41"
server_key_version: "encryptionkey@0"
I20260806 09:04:13.844549 1589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:13.869170 1589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:13.870121 1589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:13.870443 1589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:13.890726 1589 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:40679
I20260806 09:04:13.890753 1656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:40679 every 8 connection(s)
I20260806 09:04:13.892855 1589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:13.898253 1657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:13.903236 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1589
I20260806 09:04:13.903455 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:13.904318 28398 external_mini_cluster.cc:1428] Setting key 4f2bb67d81bd47814b4db44ef556f307
I20260806 09:04:13.911476 1657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b: Bootstrap starting.
I20260806 09:04:13.915547 1657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:13.916847 1657 log.cc:824] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:13.920746 1657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b: No bootstrap required, opened a new log
I20260806 09:04:13.922008 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:13.906927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.27.187.190:40679)
0806 09:04:13.908129 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:13.908142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:13.908194 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:04:13.910052 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:13.910082 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:13.910616 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:13.911110 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:13.911880 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.913975 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.916124 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:13.917022 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:13.917217 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:13.919084 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:13.919091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:13.919108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:04:13.919769 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:13.919855 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:04:13.919880 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:13.919979 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:13.920417 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 09:04:13.920879 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":453,"threads_started":1}
I20260806 09:04:13.926592 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:04:13.929178 1657 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } }
I20260806 09:04:13.929687 1657 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:13.929836 1657 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5daeafed76546eba8006d5601ce4d0b, State: Initialized, Role: FOLLOWER
I20260806 09:04:13.930514 1657 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } }
I20260806 09:04:13.930905 1657 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:13.931038 1657 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:13.931176 1657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:13.933856 1657 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } }
I20260806 09:04:13.934381 1657 leader_election.cc:302] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5daeafed76546eba8006d5601ce4d0b; no voters:
I20260806 09:04:13.935098 1657 leader_election.cc:288] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:13.935286 1662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:13.936102 1662 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [term 1 LEADER]: Becoming Leader. State: Replica: c5daeafed76546eba8006d5601ce4d0b, State: Running, Role: LEADER
I20260806 09:04:13.936744 1662 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } }
I20260806 09:04:13.937438 1657 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:13.940078 1663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5daeafed76546eba8006d5601ce4d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } } }
I20260806 09:04:13.940095 1664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5daeafed76546eba8006d5601ce4d0b. Latest consensus state: current_term: 1 leader_uuid: "c5daeafed76546eba8006d5601ce4d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5daeafed76546eba8006d5601ce4d0b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40679 } } }
I20260806 09:04:13.940577 1664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:13.940577 1663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:13.941969 1669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:13.949051 1669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:13.975122 1669 catalog_manager.cc:1347] Generated new cluster ID: 9d509bd59b1840c7abb3a5aaa9edf24a
I20260806 09:04:13.975258 1669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:13.993531 1669 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:13.995639 1669 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:14.009606 1669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5daeafed76546eba8006d5601ce4d0b: Generated new TSK 0
I20260806 09:04:14.011152 1669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:04:14.064841 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:14.034815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.27.187.190:40679)
0806 09:04:14.035091 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:14.035098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:14.035539 (+ 441us) server_negotiation.cc:407] Connection header received
0806 09:04:14.035820 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:14.035824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:14.035944 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:14.036099 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:04:14.036677 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:14.037516 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:14.039774 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:14.040272 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:14.040383 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:14.041015 (+ 632us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:04:14.063986 (+ 22971us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:04:14.064466 (+ 480us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49728: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 09:04:14.064945 1686 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:14.034665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40679 (local address 127.0.0.1:49728)
0806 09:04:14.035397 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:14.035418 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:14.035608 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:14.036151 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:14.036157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:14.036172 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:04:14.036507 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:14.036519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:14.037695 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:14.037701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:14.039553 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:14.039573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:14.040468 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:14.040475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:14.040678 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:14.040733 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:04:14.064322 (+ 23589us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:04:14.064610 (+ 288us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.187.190:40679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":344,"threads_started":1}
W20260806 09:04:14.065001 1660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49728: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260806 09:04:14.066813 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1589
I20260806 09:04:14.068193 1652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:14.168100 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1589
2026-08-06T09:04:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (825 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (444 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1602 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:16 dist-test-slave-8tdl krb5kdc[1702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:16 dist-test-slave-8tdl krb5kdc[1702](info): set up 2 sockets
Aug 06 09:04:16 dist-test-slave-8tdl krb5kdc[1702](info): commencing operation
krb5kdc: starting...
W20260806 09:04:17.358132 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:17 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:04:17.631048 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:36805
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:17.992846 1715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:17.993283 1715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:17.993451 1715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:18.001566 1715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:18.001734 1715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:18.001784 1715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:18.001924 1715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:18.002014 1715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:18.013484 1715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:36805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:18.015028 1715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:18.016559 1715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:18.029594 1723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:18.030062 1720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:18.030184 1721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:18.031054 1715 server_base.cc:1034] running on GCE node
I20260806 09:04:18.032399 1715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:18.034833 1715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:18.036131 1715 hybrid_clock.cc:648] HybridClock initialized: now 1786007058036052 us; error 60 us; skew 500 ppm
I20260806 09:04:18.036615 1715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:18 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007058, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:18.040407 1715 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:04:18.042039 1715 webserver.cc:460] Webserver started at http://127.27.187.190:36221/ using document root <none> and password file <none>
I20260806 09:04:18.042833 1715 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:18.042951 1715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:18.043221 1715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:18.046445 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "136ac55c76e649329f3f71fbada39a36"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "9e5607556ae852146618e81d35f4c630"
server_key_iv: "6460fc355d0a8ddf66cd91b693722946"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.047456 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "136ac55c76e649329f3f71fbada39a36"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "9e5607556ae852146618e81d35f4c630"
server_key_iv: "6460fc355d0a8ddf66cd91b693722946"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.053263 1715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:04:18.057412 1730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:18.059228 1715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:04:18.059475 1715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "136ac55c76e649329f3f71fbada39a36"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "9e5607556ae852146618e81d35f4c630"
server_key_iv: "6460fc355d0a8ddf66cd91b693722946"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.059695 1715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:18.072840 1715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:18.078042 1715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:18.078379 1715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:18.099291 1715 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:36805
I20260806 09:04:18.099408 1783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:36805 every 8 connection(s)
I20260806 09:04:18.100811 1715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:18.105013 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:18.108573 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1715
I20260806 09:04:18.108958 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:18.110019 28398 external_mini_cluster.cc:1428] Setting key b47c2d7f40c2783e4c32c2371fdeec1a
I20260806 09:04:18.120393 1784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36: Bootstrap starting.
I20260806 09:04:18.125285 1784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:18.127035 1784 log.cc:824] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:04:18 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:18.130901 1784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36: No bootstrap required, opened a new log
I20260806 09:04:18.137471 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:18.112421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.27.187.190:36805)
0806 09:04:18.113258 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:18.113272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:18.113315 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:04:18.115124 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:18.115151 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:18.115632 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:18.116067 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:18.118031 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.119648 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.121887 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.122734 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.122966 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:18.128553 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:18.128569 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:18.128597 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:18.128659 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:18.132295 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:18.133377 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:18.133386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:18.133393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:18.133625 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:18.134162 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:18.134167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:18.134169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:18.134937 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:18.135199 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:18.135726 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 09:04:18.136214 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":391,"threads_started":1}
I20260806 09:04:18.140880 1784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } }
I20260806 09:04:18.141386 1784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:18.141506 1784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136ac55c76e649329f3f71fbada39a36, State: Initialized, Role: FOLLOWER
I20260806 09:04:18.142134 1784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } }
I20260806 09:04:18.142377 1784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:18.142568 1784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:18.142797 1784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:18.145535 1784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } }
I20260806 09:04:18.146133 1784 leader_election.cc:302] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136ac55c76e649329f3f71fbada39a36; no voters:
I20260806 09:04:18.146806 1784 leader_election.cc:288] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:18.146974 1789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:18.147708 1789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [term 1 LEADER]: Becoming Leader. State: Replica: 136ac55c76e649329f3f71fbada39a36, State: Running, Role: LEADER
I20260806 09:04:18.148464 1789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } }
I20260806 09:04:18.149549 1784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:18.151640 1791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "136ac55c76e649329f3f71fbada39a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } } }
I20260806 09:04:18.151696 1792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136ac55c76e649329f3f71fbada39a36. Latest consensus state: current_term: 1 leader_uuid: "136ac55c76e649329f3f71fbada39a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ac55c76e649329f3f71fbada39a36" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36805 } } }
I20260806 09:04:18.152205 1791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:18.152226 1792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:18.155333 1796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:18.160386 1796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:18.170096 1796 catalog_manager.cc:1347] Generated new cluster ID: 77e5b06fa03347fba692f7086a0dfce5
I20260806 09:04:18.170248 1796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:18.183298 1796 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:18.184790 1796 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:18.196626 1796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136ac55c76e649329f3f71fbada39a36: Generated new TSK 0
I20260806 09:04:18.197686 1796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:04:18.263542 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:18.621963 1811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:18.622300 1811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:18.622400 1811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:18.630820 1811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:18.630951 1811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:18.631074 1811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:18.641808 1811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:18.643447 1811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:18.644829 1811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:18.659575 1819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:18.659531 1817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:18.659626 1816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:18.660427 1811 server_base.cc:1034] running on GCE node
I20260806 09:04:18.661096 1811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:18.662171 1811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:18.663412 1811 hybrid_clock.cc:648] HybridClock initialized: now 1786007058663361 us; error 41 us; skew 500 ppm
I20260806 09:04:18.663818 1811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:18 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007058, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:18.667457 1811 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:04:18.669147 1811 webserver.cc:460] Webserver started at http://127.27.187.129:40509/ using document root <none> and password file <none>
I20260806 09:04:18.669855 1811 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:18.669962 1811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:18.670241 1811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:18.673540 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "35b2d9ce58914a75a1d72704d9f7c452"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "a761e91b4302ca11fd5f7d14c1b81c9e"
server_key_iv: "b18766a2d09fa3e81758fe8b1e638637"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.674360 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "35b2d9ce58914a75a1d72704d9f7c452"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "a761e91b4302ca11fd5f7d14c1b81c9e"
server_key_iv: "b18766a2d09fa3e81758fe8b1e638637"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.680234 1811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 09:04:18.684092 1826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:18.685621 1811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:18.685847 1811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "35b2d9ce58914a75a1d72704d9f7c452"
format_stamp: "Formatted at 2026-08-06 09:04:18 on dist-test-slave-8tdl"
server_key: "a761e91b4302ca11fd5f7d14c1b81c9e"
server_key_iv: "b18766a2d09fa3e81758fe8b1e638637"
server_key_version: "encryptionkey@0"
I20260806 09:04:18.686086 1811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:18.713541 1811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:18.718180 1811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:18.718500 1811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:18.719564 1811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:18.721087 1811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:18.721170 1811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:18.721328 1811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:18.721395 1811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:18.757270 1811 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:46135
I20260806 09:04:18.757381 1943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:46135 every 8 connection(s)
I20260806 09:04:18.758657 1811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:18.761209 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1811
I20260806 09:04:18.761443 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:18.762312 28398 external_mini_cluster.cc:1428] Setting key 8d4bc3316928e03bd775573eeb9236b4
Aug 06 09:04:18 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007058, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:18.785245 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:18.761815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:40303 (local address 127.27.187.190:36805)
0806 09:04:18.762398 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:18.762406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:18.763741 (+ 1335us) server_negotiation.cc:407] Connection header received
0806 09:04:18.766551 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:18.766557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:18.766644 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:18.766825 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:18.769260 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.770058 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.772831 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.773341 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.773455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:18.778121 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:18.778127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:18.778130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:18.778170 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:18.781103 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:18.782367 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:18.782372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:18.782374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:18.782515 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:18.783132 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:18.783136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:18.783138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:18.783486 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:18.783608 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:18.784732 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0806 09:04:18.784973 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
I20260806 09:04:18.787148 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:18.762564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36805 (local address 127.27.187.129:40303)
0806 09:04:18.763573 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:18.763623 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:18.766090 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:18.767164 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:18.767181 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:18.768064 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:18.768991 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:18.769023 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.770306 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.770316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:18.772627 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:18.772648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:18.773603 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:18.773613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:18.773970 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:18.774947 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:18.774972 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:18.777796 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:18.781329 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:18.781344 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:18.781371 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:18.782168 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:18.782643 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:18.782650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:18.782656 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:18.782979 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:18.783644 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:18.783655 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:18.784191 (+ 536us) client_negotiation.cc:769] Sending connection context
0806 09:04:18.784597 (+ 406us) client_negotiation.cc:240] Negotiation successful
0806 09:04:18.785299 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":389,"threads_started":1}
I20260806 09:04:18.789311 1944 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36805
I20260806 09:04:18.789736 1944 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:04:18.790843 1944 heartbeater.cc:511] Master 127.27.187.190:36805 requested a full tablet report, sending...
I20260806 09:04:18.793520 1748 ts_manager.cc:194] Registered new tserver with Master: 35b2d9ce58914a75a1d72704d9f7c452 (127.27.187.129:46135)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:04:18.795862 1748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:40303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:04:18.828085 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:19.198212 1950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:19.198580 1950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:19.198804 1950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:19.206465 1950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:19.206650 1950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:19.206872 1950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:19.217344 1950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:19.218878 1950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:19.220297 1950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:19.235148 1958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:19.235178 1955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:19.235219 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:19.236143 1950 server_base.cc:1034] running on GCE node
I20260806 09:04:19.236832 1950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:19.237818 1950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:19.239032 1950 hybrid_clock.cc:648] HybridClock initialized: now 1786007059238992 us; error 38 us; skew 500 ppm
I20260806 09:04:19.239348 1950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:19.242966 1950 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:04:19.244700 1950 webserver.cc:460] Webserver started at http://127.27.187.130:34573/ using document root <none> and password file <none>
I20260806 09:04:19.245468 1950 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:19.245622 1950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:19.245995 1950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:19.249385 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "75bb38329fdd49929929bc83d7465bea"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "d12dc36958ea30938268950fbbf5d7c4"
server_key_iv: "987102f9708f12a61f74402dabeb542d"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.250376 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "75bb38329fdd49929929bc83d7465bea"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "d12dc36958ea30938268950fbbf5d7c4"
server_key_iv: "987102f9708f12a61f74402dabeb542d"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.256625 1950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:04:19.260699 1965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.262264 1950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:04:19.262446 1950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "75bb38329fdd49929929bc83d7465bea"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "d12dc36958ea30938268950fbbf5d7c4"
server_key_iv: "987102f9708f12a61f74402dabeb542d"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.262665 1950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:19.286782 1950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:19.291792 1950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:19.292140 1950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:19.293251 1950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:19.294778 1950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:19.294883 1950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.295073 1950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:19.295126 1950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.332348 1950 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:44229
I20260806 09:04:19.332479 2079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:44229 every 8 connection(s)
I20260806 09:04:19.333801 1950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:19.337368 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 1950
I20260806 09:04:19.337561 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:19.338034 28398 external_mini_cluster.cc:1428] Setting key fb07e94372c01ab9a842bf2591dffdee
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:19.358767 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:19.336640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:40885 (local address 127.27.187.190:36805)
0806 09:04:19.337134 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:19.337141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:19.338428 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 09:04:19.340722 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:19.340727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:19.340815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:19.340943 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:19.342588 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.343825 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.345491 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.346280 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.346439 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:19.350783 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:19.350789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:19.350793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:19.350831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:19.354773 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:19.355844 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:19.355851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:19.355855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:19.356023 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:19.356568 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:19.356574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:19.356578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:19.356909 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:19.357090 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:19.357970 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 09:04:19.358286 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":231,"threads_started":1}
I20260806 09:04:19.359948 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:19.337310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36805 (local address 127.27.187.130:40885)
0806 09:04:19.338238 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:19.338286 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:19.340423 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:19.341160 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:19.341169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:19.341766 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:19.342382 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:19.342408 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.343997 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.344001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:19.345316 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:19.345330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.346539 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.346545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:19.346862 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:19.347672 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:19.347705 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:19.350454 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:19.354997 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:19.355009 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:19.355039 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:19.355678 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:19.356164 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:19.356171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:19.356175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:19.356429 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:19.357062 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:19.357072 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:19.357535 (+ 463us) client_negotiation.cc:769] Sending connection context
0806 09:04:19.357871 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 09:04:19.358523 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":438,"threads_started":1}
I20260806 09:04:19.361937 2080 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36805
I20260806 09:04:19.362396 2080 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
I20260806 09:04:19.363487 2080 heartbeater.cc:511] Master 127.27.187.190:36805 requested a full tablet report, sending...
I20260806 09:04:19.365617 1748 ts_manager.cc:194] Registered new tserver with Master: 75bb38329fdd49929929bc83d7465bea (127.27.187.130:44229)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
I20260806 09:04:19.366621 1748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:40885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:04:19.402222 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:19.772408 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:19.772727 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:19.772825 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:19.780441 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:19.780575 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:19.780691 2087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:19.791079 2087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:19.792471 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:19.793807 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:04:19.799743 1944 heartbeater.cc:503] Master 127.27.187.190:36805 was elected leader, sending a full tablet report...
W20260806 09:04:19.809631 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:19.809778 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:19.809669 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:19.811079 2087 server_base.cc:1034] running on GCE node
I20260806 09:04:19.811878 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:19.813122 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:19.814390 2087 hybrid_clock.cc:648] HybridClock initialized: now 1786007059814348 us; error 40 us; skew 500 ppm
I20260806 09:04:19.814831 2087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:19.818464 2087 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:04:19.820199 2087 webserver.cc:460] Webserver started at http://127.27.187.131:37633/ using document root <none> and password file <none>
I20260806 09:04:19.820937 2087 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:19.821055 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:19.821344 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:19.824537 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "2e391a0339ca41b899445f40db965194"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "7bd746b52fcb3103877c365f12bb7aef"
server_key_iv: "60d2b22e23db2365272bbddff1ce0483"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.825374 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "2e391a0339ca41b899445f40db965194"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "7bd746b52fcb3103877c365f12bb7aef"
server_key_iv: "60d2b22e23db2365272bbddff1ce0483"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.831604 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:04:19.836031 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.837815 2087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260806 09:04:19.837986 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "2e391a0339ca41b899445f40db965194"
format_stamp: "Formatted at 2026-08-06 09:04:19 on dist-test-slave-8tdl"
server_key: "7bd746b52fcb3103877c365f12bb7aef"
server_key_iv: "60d2b22e23db2365272bbddff1ce0483"
server_key_version: "encryptionkey@0"
I20260806 09:04:19.838219 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:19.857187 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:19.862905 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:19.863252 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:19.864291 2087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:19.865674 2087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:19.865754 2087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.865847 2087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:19.865892 2087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:19.903244 2087 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45933
I20260806 09:04:19.903330 2216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45933 every 8 connection(s)
I20260806 09:04:19.904605 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:19.910550 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2087
I20260806 09:04:19.910892 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:19.911669 28398 external_mini_cluster.cc:1428] Setting key 51fd6c9f05e11b29ad561c75389150c5
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:19.930037 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:19.907482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:53245 (local address 127.27.187.190:36805)
0806 09:04:19.908068 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:19.908074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:19.909367 (+ 1293us) server_negotiation.cc:407] Connection header received
0806 09:04:19.912608 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:19.912615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:19.912729 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:19.912875 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:19.915004 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.916034 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.917849 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.918305 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.918459 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:19.922753 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:19.922760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:19.922764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:19.922806 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:19.925920 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:19.927266 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:19.927275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:19.927278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:19.927452 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:19.927967 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:19.927977 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:19.927979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:19.928365 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:19.928537 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:19.929520 (+ 983us) server_negotiation.cc:299] Negotiation successful
0806 09:04:19.929716 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":306,"threads_started":1}
I20260806 09:04:19.931113 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:19.908156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36805 (local address 127.27.187.131:53245)
0806 09:04:19.909084 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:19.909137 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:19.912159 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:19.913179 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:19.913189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:19.914035 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:19.914807 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:19.914830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.916235 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.916240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:19.917671 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:19.917685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.918572 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.918580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:19.918924 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:19.919722 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:19.919748 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:19.922435 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:19.926190 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:19.926199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:19.926218 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:19.927086 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:19.927585 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:19.927592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:19.927600 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:19.927829 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:19.928552 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:19.928564 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:19.929070 (+ 506us) client_negotiation.cc:769] Sending connection context
0806 09:04:19.929375 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 09:04:19.929823 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":391,"threads_started":1}
I20260806 09:04:19.932997 2217 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36805
I20260806 09:04:19.933430 2217 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:19.934290 2217 heartbeater.cc:511] Master 127.27.187.190:36805 requested a full tablet report, sending...
I20260806 09:04:19.936056 1748 ts_manager.cc:194] Registered new tserver with Master: 2e391a0339ca41b899445f40db965194 (127.27.187.131:45933)
I20260806 09:04:19.937049 1748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:53245
I20260806 09:04:19.941655 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/test-admin.keytab.
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:19.977481 28398 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 09:04:19.983197 2225 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:19.980516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.27.187.129:46135)
0806 09:04:19.981141 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:19.981147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:19.981177 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:04:19.981293 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:19.981301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:19.981533 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:19.981692 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:19.982790 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40642: BlockingRecv error: recv got EOF from 127.0.0.1:40642 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":298,"threads_started":1}
W20260806 09:04:19.983318 28398 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:19.985369 28398 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:19.986964 28398 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:04:19 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:20.007264 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:19.988977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.27.187.129:46135)
0806 09:04:19.989212 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:19.989219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:19.989431 (+ 212us) server_negotiation.cc:407] Connection header received
0806 09:04:19.989695 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:19.989699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:19.989767 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:19.989885 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:19.991076 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.992210 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.993618 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:19.994287 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:19.994448 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:19.999240 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:19.999265 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:19.999282 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:19.999332 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.003634 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.004594 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.004602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.004605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.004770 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.005251 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.005268 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.005272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.005708 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.006059 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.006569 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.006927 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 09:04:20.015138 2225 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:20.012795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.27.187.129:46135)
0806 09:04:20.013185 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.013192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.013376 (+ 184us) server_negotiation.cc:407] Connection header received
0806 09:04:20.013661 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.013666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.013734 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.013851 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.014788 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40654: BlockingRecv error: recv got EOF from 127.0.0.1:40654 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260806 09:04:20.015760 28398 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc)
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:20.017622 28398 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:20.019922 28398 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:20.036464 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:20.021956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.27.187.129:46135)
0806 09:04:20.022196 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.022201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.022319 (+ 118us) server_negotiation.cc:407] Connection header received
0806 09:04:20.022520 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.022525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.022600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.022792 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.023849 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.024687 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.025908 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.026398 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.026513 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:20.030642 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:20.030647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:20.030651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:20.030780 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.033777 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.034507 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.034511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.034514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.034666 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.035094 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.035097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.035099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.035446 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.035643 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.035886 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.036060 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 09:04:20.042023 2225 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:20.039914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.27.187.129:46135)
0806 09:04:20.040129 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.040136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.040346 (+ 210us) server_negotiation.cc:407] Connection header received
0806 09:04:20.040503 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.040507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.040583 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.040669 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.041731 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40672: BlockingRecv error: recv got EOF from 127.0.0.1:40672 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 09:04:20.042136 28398 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007003447047-28398-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:20.044034 28398 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:20.045704 28398 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:20.063740 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:20.047706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.27.187.129:46135)
0806 09:04:20.047965 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.047972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.048175 (+ 203us) server_negotiation.cc:407] Connection header received
0806 09:04:20.048465 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.048471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.048549 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.048664 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.049692 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.050598 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.052004 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.052484 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.052628 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:20.057337 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:20.057348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:20.057352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:20.057394 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.060616 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.061473 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.061478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.061482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.061669 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.062109 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.062115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.062118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.062465 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.062841 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.063116 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.063385 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:04:20.065706 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1811
I20260806 09:04:20.079821 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:20.066778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.27.187.129:46135)
0806 09:04:20.067050 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.067058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.067386 (+ 328us) server_negotiation.cc:407] Connection header received
0806 09:04:20.067680 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.067684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.067747 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.067860 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.068896 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.069628 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.071009 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.071454 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.071606 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:20.073948 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:20.073953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:20.073957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:20.073995 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.076880 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.077644 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.077650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.077654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.077823 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.078224 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.078230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.078234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.078541 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.078885 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.079198 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.079456 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:04:20.080503 1937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:20.171406 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1811
I20260806 09:04:20.192883 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1950
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:04:20.212836 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:20.194135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.27.187.130:44229)
0806 09:04:20.194837 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.194845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.194880 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:04:20.194956 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.194963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.195207 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.195369 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.196649 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.197723 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.199570 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.200221 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.200346 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:20.205231 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:20.205251 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:20.205268 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:20.205317 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.209040 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.209982 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.209991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.209993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.210294 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.210947 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.210952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.210957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.211455 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.211821 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.212186 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.212495 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":324,"threads_started":1}
I20260806 09:04:20.214308 2075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:20.302654 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1950
I20260806 09:04:20.321380 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2087
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[1702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:04:20.340938 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:20.322576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.27.187.131:45933)
0806 09:04:20.323360 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:20.323368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:20.323401 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:04:20.323466 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:20.323514 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:20.323752 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:20.323912 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:20.325219 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.326287 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.328292 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:20.328943 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:20.329056 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:20.333597 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:20.333613 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:20.333630 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:20.333677 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:20.337265 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.338473 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.338482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.338484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.338635 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:20.339162 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:20.339165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:20.339167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:20.339627 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:20.339881 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:20.340254 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 09:04:20.340636 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":429,"threads_started":1}
I20260806 09:04:20.342545 2212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:20.429148 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2087
I20260806 09:04:20.446997 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 1715
I20260806 09:04:20.448493 1779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:20.558861 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 1715
2026-08-06T09:04:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4351 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[2245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[2245](info): set up 2 sockets
Aug 06 09:04:20 dist-test-slave-8tdl krb5kdc[2245](info): commencing operation
krb5kdc: starting...
W20260806 09:04:21.703277 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:21 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab.
Entry for principal oryx/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab.
I20260806 09:04:21.992547 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38867
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab
--principal=oryx/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:22.346133 2259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:22.346463 2259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:22.346621 2259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:22.354206 2259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:22.354344 2259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:22.354434 2259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:22.354521 2259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:22.354588 2259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:22.365468 2259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab
--principal=oryx/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:22.366994 2259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:22.368417 2259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:22.381312 2264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:22.381361 2267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:22.381269 2265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:22.382460 2259 server_base.cc:1034] running on GCE node
I20260806 09:04:22.383435 2259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:22.385138 2259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:22.386456 2259 hybrid_clock.cc:648] HybridClock initialized: now 1786007062386417 us; error 41 us; skew 500 ppm
I20260806 09:04:22.386907 2259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:22 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007062, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:22.394377 2259 init.cc:377] Logged in from keytab as oryx/127.27.187.190@KRBTEST.COM (short username oryx)
I20260806 09:04:22.396466 2259 webserver.cc:460] Webserver started at http://127.27.187.190:43799/ using document root <none> and password file <none>
I20260806 09:04:22.397261 2259 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:22.397426 2259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:22.397823 2259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:22.401142 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "9115f4af30864fbf9fcce0a985db9c4e"
format_stamp: "Formatted at 2026-08-06 09:04:22 on dist-test-slave-8tdl"
server_key: "ba3b46afbee3af1c13535331c939dce9"
server_key_iv: "2a3f37b9858b07c53370951834450220"
server_key_version: "encryptionkey@0"
I20260806 09:04:22.402012 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "9115f4af30864fbf9fcce0a985db9c4e"
format_stamp: "Formatted at 2026-08-06 09:04:22 on dist-test-slave-8tdl"
server_key: "ba3b46afbee3af1c13535331c939dce9"
server_key_iv: "2a3f37b9858b07c53370951834450220"
server_key_version: "encryptionkey@0"
I20260806 09:04:22.408916 2259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 09:04:22.412838 2274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:22.414413 2259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:04:22.414638 2259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "9115f4af30864fbf9fcce0a985db9c4e"
format_stamp: "Formatted at 2026-08-06 09:04:22 on dist-test-slave-8tdl"
server_key: "ba3b46afbee3af1c13535331c939dce9"
server_key_iv: "2a3f37b9858b07c53370951834450220"
server_key_version: "encryptionkey@0"
I20260806 09:04:22.414953 2259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:22.436781 2259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:22.441810 2259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:22.442261 2259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:22.462400 2259 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38867
I20260806 09:04:22.462433 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38867 every 8 connection(s)
I20260806 09:04:22.463932 2259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:22.468024 2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:22.472388 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2259
I20260806 09:04:22.472617 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:22.473480 28398 external_mini_cluster.cc:1428] Setting key 90116c8594c985363979791be313f6c3
I20260806 09:04:22.480587 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Bootstrap starting.
I20260806 09:04:22.483932 2328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:22.485020 2328 log.cc:824] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:22.489236 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: No bootstrap required, opened a new log
Aug 06 09:04:22 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:22.498785 2328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } }
I20260806 09:04:22.499418 2328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:22.499583 2328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9115f4af30864fbf9fcce0a985db9c4e, State: Initialized, Role: FOLLOWER
I20260806 09:04:22.500368 2328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } }
I20260806 09:04:22.500655 2328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:22.500820 2328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:22.500970 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:22.475959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.27.187.190:38867)
0806 09:04:22.476816 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:22.476828 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:22.476872 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:04:22.479277 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:22.479303 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:22.479743 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:22.480169 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:22.481799 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:22.483288 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:22.484720 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:22.485579 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:22.485804 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:22.490999 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:22.491008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:22.491029 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:22.491075 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:22.494593 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:22.495579 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:22.495625 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:22.495635 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:22.495930 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:22.496562 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:22.496586 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:22.496595 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:22.497595 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:22.497949 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:22.499016 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 09:04:22.499520 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":381,"threads_started":1}
I20260806 09:04:22.501025 2328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:22.504385 2328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } }
I20260806 09:04:22.504957 2328 leader_election.cc:302] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9115f4af30864fbf9fcce0a985db9c4e; no voters:
I20260806 09:04:22.505630 2328 leader_election.cc:288] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:22.505816 2333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:22.506604 2333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 LEADER]: Becoming Leader. State: Replica: 9115f4af30864fbf9fcce0a985db9c4e, State: Running, Role: LEADER
I20260806 09:04:22.507232 2333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } }
I20260806 09:04:22.507905 2328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:22.510159 2334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } }
I20260806 09:04:22.510571 2334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:22.511871 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115f4af30864fbf9fcce0a985db9c4e. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } }
I20260806 09:04:22.512588 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:22.516826 2338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:22.522416 2338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:22.533933 2338 catalog_manager.cc:1347] Generated new cluster ID: fc6ce6ad38d34352a7088f73bd0a7208
I20260806 09:04:22.534076 2338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:22.560643 2338 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:22.562278 2338 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:22.569546 2338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Generated new TSK 0
I20260806 09:04:22.570499 2338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
I20260806 09:04:22.651577 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:23.033150 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:23.033463 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:23.033627 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:23.041340 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:23.041467 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:23.041692 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:23.052034 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:23.053515 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:23.054988 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:23.069233 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:23.069418 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:23.069422 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:23.070223 2355 server_base.cc:1034] running on GCE node
I20260806 09:04:23.071038 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:23.072319 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:23.073556 2355 hybrid_clock.cc:648] HybridClock initialized: now 1786007063073503 us; error 43 us; skew 500 ppm
I20260806 09:04:23.073943 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:23 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007063, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:23.077749 2355 init.cc:377] Logged in from keytab as oryx/127.27.187.129@KRBTEST.COM (short username oryx)
I20260806 09:04:23.079577 2355 webserver.cc:460] Webserver started at http://127.27.187.129:36809/ using document root <none> and password file <none>
I20260806 09:04:23.080353 2355 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:23.080513 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:23.080834 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:23.084124 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "3566399ca52e4a689558c05369c24580"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "a232bce24916ecc5061df9de7c60ee0d"
server_key_iv: "4935371ef941ee9859f5244c37ed0606"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.085007 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "3566399ca52e4a689558c05369c24580"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "a232bce24916ecc5061df9de7c60ee0d"
server_key_iv: "4935371ef941ee9859f5244c37ed0606"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.090838 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:04:23.095039 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.096705 2355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:23.096921 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "3566399ca52e4a689558c05369c24580"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "a232bce24916ecc5061df9de7c60ee0d"
server_key_iv: "4935371ef941ee9859f5244c37ed0606"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.097175 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:23.120491 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:23.125548 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:23.125885 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:23.126996 2355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:23.128501 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:23.128616 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.128763 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:23.128835 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.165993 2355 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:46827
I20260806 09:04:23.166047 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:46827 every 8 connection(s)
I20260806 09:04:23.167444 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:23.174458 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2355
I20260806 09:04:23.174739 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:23.175516 28398 external_mini_cluster.cc:1428] Setting key 881896c8633cc6ef2c37d3f4564ac427
Aug 06 09:04:23 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007063, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:23.191548 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:23.170473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34797 (local address 127.27.187.190:38867)
0806 09:04:23.171022 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:23.171030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:23.172138 (+ 1108us) server_negotiation.cc:407] Connection header received
0806 09:04:23.174280 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:23.174287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:23.174412 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:23.174560 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:23.176800 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.177555 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.179357 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.179788 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.179932 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:23.184455 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:23.184462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:23.184465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:23.184506 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:23.187700 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:23.188715 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:23.188720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:23.188723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:23.188871 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:23.189426 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:23.189430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:23.189433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:23.189754 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:23.189939 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:23.190840 (+ 901us) server_negotiation.cc:299] Negotiation successful
0806 09:04:23.191118 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":268,"threads_started":1}
I20260806 09:04:23.192695 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:23.171087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:34797)
0806 09:04:23.171946 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:23.171989 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:23.173950 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:23.175071 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:23.175081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:23.175806 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:23.176590 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:23.176621 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.177741 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.177751 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:23.179167 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:23.179189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.180059 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.180069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:23.180438 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:23.181261 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:23.181293 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:23.184083 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:23.187913 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:23.187924 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:23.187947 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:23.188523 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:23.189010 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:23.189016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:23.189020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:23.189271 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:23.189908 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:23.189918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:23.190352 (+ 434us) client_negotiation.cc:769] Sending connection context
0806 09:04:23.190663 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 09:04:23.191349 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":328,"threads_started":1}
I20260806 09:04:23.194588 2485 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:23.195010 2485 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:23.195820 2485 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:23.198596 2292 ts_manager.cc:194] Registered new tserver with Master: 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827)
I20260806 09:04:23.200974 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:34797
WARNING: no policy specified for oryx/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
I20260806 09:04:23.239382 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:23.601645 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:23.602015 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:23.602131 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:23.610077 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:23.610215 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:23.610335 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:23.620853 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:23.622363 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:23.623709 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:23.637954 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:23.637988 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:23.638022 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:23.638974 2492 server_base.cc:1034] running on GCE node
I20260806 09:04:23.639609 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:23.640618 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:23.641932 2492 hybrid_clock.cc:648] HybridClock initialized: now 1786007063641853 us; error 68 us; skew 500 ppm
I20260806 09:04:23.642395 2492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:23 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007063, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:23.646343 2492 init.cc:377] Logged in from keytab as oryx/127.27.187.130@KRBTEST.COM (short username oryx)
I20260806 09:04:23.648057 2492 webserver.cc:460] Webserver started at http://127.27.187.130:35287/ using document root <none> and password file <none>
I20260806 09:04:23.648841 2492 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:23.648985 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:23.649338 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:23.652549 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "f3a3b54ab59840a6a40662b091b81776"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "81d129408a8b9ac48019b3c1eb8b385b"
server_key_iv: "5be51527b51d8e0d3a6f4842fcd26b18"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.653424 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "f3a3b54ab59840a6a40662b091b81776"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "81d129408a8b9ac48019b3c1eb8b385b"
server_key_iv: "5be51527b51d8e0d3a6f4842fcd26b18"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.659423 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:04:23.663460 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.665278 2492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:23.665496 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "f3a3b54ab59840a6a40662b091b81776"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "81d129408a8b9ac48019b3c1eb8b385b"
server_key_iv: "5be51527b51d8e0d3a6f4842fcd26b18"
server_key_version: "encryptionkey@0"
I20260806 09:04:23.665728 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:23.689186 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:23.694528 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:23.694919 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:23.696102 2492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:23.697791 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:23.697880 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.698016 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:23.698076 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:23.743319 2492 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:35413
I20260806 09:04:23.743338 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:35413 every 8 connection(s)
I20260806 09:04:23.745291 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:23.751072 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2492
I20260806 09:04:23.751284 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:23.751938 28398 external_mini_cluster.cc:1428] Setting key abfb036aa0a1b0eeaa3399ebc1a11271
Aug 06 09:04:23 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007063, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:23.775840 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:23.750806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:54391 (local address 127.27.187.190:38867)
0806 09:04:23.755152 (+ 4346us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:23.755158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:23.755180 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:04:23.755245 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:23.755250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:23.755337 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:23.755474 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:23.758409 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.759454 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.762059 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.762584 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.762828 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:23.767330 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:23.767341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:23.767349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:23.767402 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:23.771642 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:23.772776 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:23.772786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:23.772792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:23.772973 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:23.773471 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:23.773478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:23.773484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:23.773935 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:23.774118 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:23.774982 (+ 864us) server_negotiation.cc:299] Negotiation successful
0806 09:04:23.775271 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4209,"thread_start_us":4032,"threads_started":1}
I20260806 09:04:23.776690 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:23.750033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:54391)
0806 09:04:23.751234 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:23.751297 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:23.754204 (+ 2907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:23.755934 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:23.755951 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:23.757019 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:23.758148 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:23.758181 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.759677 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.759688 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:23.761850 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:23.761870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:23.762914 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:23.762924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:23.763259 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:23.764084 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:23.764117 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:23.766950 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:23.771942 (+ 4992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:23.771952 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:23.771969 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:23.772596 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:23.773107 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:23.773110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:23.773112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:23.773344 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:23.774095 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:23.774101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:23.774494 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 09:04:23.774860 (+ 366us) client_negotiation.cc:240] Negotiation successful
0806 09:04:23.775341 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":414,"threads_started":1}
I20260806 09:04:23.778654 2623 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:23.779069 2623 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:23.779980 2623 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:23.781818 2292 ts_manager.cc:194] Registered new tserver with Master: f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
WARNING: no policy specified for oryx/127.27.187.131@KRBTEST.COM; defaulting to no policy
I20260806 09:04:23.782930 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:54391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
I20260806 09:04:23.819298 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:24.201221 2630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:24.201597 2630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:24.201704 2630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:04:24.204758 2485 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
W20260806 09:04:24.210273 2630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:24.210399 2630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:24.210628 2630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:24.221455 2630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:24.222975 2630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:24.224402 2630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:24.239291 2638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:24.239393 2635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:24.239393 2636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:24.240551 2630 server_base.cc:1034] running on GCE node
I20260806 09:04:24.241267 2630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:24.242354 2630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:24.243646 2630 hybrid_clock.cc:648] HybridClock initialized: now 1786007064243570 us; error 91 us; skew 500 ppm
I20260806 09:04:24.244007 2630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007064, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:24.247663 2630 init.cc:377] Logged in from keytab as oryx/127.27.187.131@KRBTEST.COM (short username oryx)
I20260806 09:04:24.249563 2630 webserver.cc:460] Webserver started at http://127.27.187.131:37695/ using document root <none> and password file <none>
I20260806 09:04:24.250268 2630 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:24.250376 2630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:24.250645 2630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:24.253796 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "dd3df5bb5d7b8a732c3bbae5558ad403"
server_key_iv: "92823554b57518509ca44a888839fbcb"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.254598 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "dd3df5bb5d7b8a732c3bbae5558ad403"
server_key_iv: "92823554b57518509ca44a888839fbcb"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.260245 2630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:04:24.264196 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:24.265736 2630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:24.265975 2630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "dd3df5bb5d7b8a732c3bbae5558ad403"
server_key_iv: "92823554b57518509ca44a888839fbcb"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.266316 2630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:24.285646 2630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:24.290376 2630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:24.290767 2630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:24.291863 2630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:24.293417 2630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:24.293493 2630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:24.293655 2630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:24.293707 2630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:24.330963 2630 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45661
I20260806 09:04:24.331027 2759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45661 every 8 connection(s)
I20260806 09:04:24.332376 2630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:24.338610 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2630
I20260806 09:04:24.338874 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:24.339324 28398 external_mini_cluster.cc:1428] Setting key f717df917751a059061190cf7fa0fe29
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007064, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:24.356061 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.335317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:56063 (local address 127.27.187.190:38867)
0806 09:04:24.335822 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:24.335826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:24.336836 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 09:04:24.339637 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:24.339643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:24.339759 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:24.339887 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:24.341637 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.342607 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.344306 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.344742 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.344853 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.348846 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:24.348852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:24.348856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:24.348895 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:24.352003 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.352915 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.352921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.352923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.353073 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.353785 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.353790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.353793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.354258 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:24.354485 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:24.355375 (+ 890us) server_negotiation.cc:299] Negotiation successful
0806 09:04:24.355647 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260806 09:04:24.357095 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.335855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:56063)
0806 09:04:24.336659 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:24.336698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:24.339264 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:24.340164 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:24.340173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:24.340864 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:24.341472 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.341491 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.342815 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.342823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.344131 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.344145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.345001 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.345006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.345241 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.345984 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:24.346014 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:24.348530 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:24.352196 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.352204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.352228 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.352724 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.353306 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.353311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.353314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.353565 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.354423 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:24.354434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:24.354942 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 09:04:24.355240 (+ 298us) client_negotiation.cc:240] Negotiation successful
0806 09:04:24.355902 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":356,"threads_started":1}
I20260806 09:04:24.358744 2760 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:24.359030 2760 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:24.359794 2760 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:24.361403 2292 ts_manager.cc:194] Registered new tserver with Master: 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661)
I20260806 09:04:24.362375 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:56063
I20260806 09:04:24.368393 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.187.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.189.keytab.
Entry for principal oryx/127.27.187.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.189.keytab.
I20260806 09:04:24.430902 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:34301
--webserver_interface=127.27.187.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.189.keytab
--principal=oryx/127.27.187.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38867,127.27.187.189:34301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:24.784413 2766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:24.784798 2766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:24.784992 2766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:04:24.786386 2623 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
W20260806 09:04:24.793125 2766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:24.793308 2766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:24.793424 2766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:24.793524 2766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:24.793610 2766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:24.804176 2766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38867,127.27.187.189:34301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.189.keytab
--principal=oryx/127.27.187.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:34301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.187.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
W20260806 09:04:24.804638 2766 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.187.190:38867,127.27.187.189:34301'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 09:04:24.805768 2766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:24.807214 2766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:24.821383 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:24.821318 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:24.821290 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:24.822335 2766 server_base.cc:1034] running on GCE node
I20260806 09:04:24.823298 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:24.824919 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:24.826231 2766 hybrid_clock.cc:648] HybridClock initialized: now 1786007064826182 us; error 67 us; skew 500 ppm
I20260806 09:04:24.826759 2766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007064, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:24.830396 2766 init.cc:377] Logged in from keytab as oryx/127.27.187.189@KRBTEST.COM (short username oryx)
I20260806 09:04:24.832212 2766 webserver.cc:460] Webserver started at http://127.27.187.189:35491/ using document root <none> and password file <none>
I20260806 09:04:24.833005 2766 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:24.833153 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:24.833514 2766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:24.837041 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/instance:
uuid: "5f143c3e47bf49fba124d1aef220f7b9"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "89c210c83b4f1cfd7343ec4a649432bb"
server_key_iv: "bbec34287d373fd6801d8e937a7eaad3"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.837950 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal/instance:
uuid: "5f143c3e47bf49fba124d1aef220f7b9"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "89c210c83b4f1cfd7343ec4a649432bb"
server_key_iv: "bbec34287d373fd6801d8e937a7eaad3"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.844085 2766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:04:24.848191 2781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:24.849893 2766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:24.850108 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "5f143c3e47bf49fba124d1aef220f7b9"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "89c210c83b4f1cfd7343ec4a649432bb"
server_key_iv: "bbec34287d373fd6801d8e937a7eaad3"
server_key_version: "encryptionkey@0"
I20260806 09:04:24.850389 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:24.880709 2766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:24.885668 2766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:24.885977 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:24.905718 2766 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.189:34301
I20260806 09:04:24.905757 2832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.189:34301 every 8 connection(s)
I20260806 09:04:24.907358 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
I20260806 09:04:24.909409 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2766
I20260806 09:04:24.912492 2833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:24.922238 2833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007064, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.189@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:24.936029 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.911788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.27.187.189:34301)
0806 09:04:24.912729 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:24.912747 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:24.912814 (+ 67us) server_negotiation.cc:407] Connection header received
0806 09:04:24.914490 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:24.914514 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:24.915022 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:24.915386 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:24.917442 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.919120 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.921085 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.921984 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.922141 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.927952 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:24.927963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:24.927974 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:24.928017 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:24.931392 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.932203 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.932211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.932217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.932409 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.932862 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.932868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.932871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.933476 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:24.933815 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:24.934313 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 09:04:24.934858 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260806 09:04:24.941958 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.926049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.27.187.190:38867)
0806 09:04:24.926643 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:24.926650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:24.927249 (+ 599us) server_negotiation.cc:407] Connection header received
0806 09:04:24.927488 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:24.927492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:24.927561 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:24.927657 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:24.929083 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.929878 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.931499 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.931929 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.932098 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.935794 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:24.935800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:24.935803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:24.935845 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:24.938769 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.939724 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.939730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.939734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.939875 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.940286 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.940291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.940294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.940650 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:24.940855 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:24.941319 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 09:04:24.941567 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":272,"threads_started":1}
I20260806 09:04:24.941995 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.926044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38516)
0806 09:04:24.927062 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:24.927126 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:24.927331 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:24.927777 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:24.927784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:24.928627 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:24.928908 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.928918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.930042 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.930053 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.931338 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.931352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.932149 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.932156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.932348 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.933086 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:24.933113 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:24.935525 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:24.938927 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.938939 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.938946 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.939578 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.939997 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.940002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.940006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.940162 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.940812 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:24.940820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:24.941006 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 09:04:24.941202 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 09:04:24.941559 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":534,"threads_started":1}
I20260806 09:04:24.944074 2833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:04:24 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007064, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.189@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
I20260806 09:04:24.958359 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.944767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56674)
0806 09:04:24.945002 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:24.945021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:24.945183 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:24.945534 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:24.945538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:24.945903 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:24.946051 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.946060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.947249 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.947253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.948253 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:24.948265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.949058 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.949063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:24.949235 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.949925 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:24.949943 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:24.952126 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:24.955830 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.955834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.955838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.956327 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.956724 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:24.956727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:24.956730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:24.956858 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:24.957514 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:24.957517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:24.957655 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 09:04:24.957812 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 09:04:24.958061 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:04:24.958819 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:24.944976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.27.187.189:34301)
0806 09:04:24.945135 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:24.945139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:24.945155 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:04:24.945347 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:24.945351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:24.945410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:24.945488 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:24.946267 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.947088 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.948424 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:24.948889 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:24.949004 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:24.952335 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:24.952340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:24.952343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:24.952375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:24.955663 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.956467 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.956471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.956474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.956612 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:24.956999 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:24.957003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:24.957005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:24.957382 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:24.957512 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:24.957922 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 09:04:24.958299 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":53}
I20260806 09:04:24.963632 2833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Bootstrap starting.
I20260806 09:04:24.966899 2833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:24.967913 2833 log.cc:824] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:24.971386 2833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: No bootstrap required, opened a new log
I20260806 09:04:24.978539 2833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } }
I20260806 09:04:24.979000 2833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:24.979100 2833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f143c3e47bf49fba124d1aef220f7b9, State: Initialized, Role: FOLLOWER
I20260806 09:04:24.979771 2833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } }
I20260806 09:04:24.980741 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } } }
I20260806 09:04:24.981182 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:24.981910 2833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:04:24.992357 2851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:04:24.992532 2851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:04:24.996528 2766 master_runner.cc:428] Detected that this master 5f143c3e47bf49fba124d1aef220f7b9 is joining an existing cluster
I20260806 09:04:24.996649 2766 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.187.189:34301
I20260806 09:04:25.021687 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.002628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.27.187.190:38867)
0806 09:04:25.002837 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.002844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.002929 (+ 85us) server_negotiation.cc:407] Connection header received
0806 09:04:25.003266 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.003273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.003380 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.003463 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.005183 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.006345 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.008593 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.009326 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.009444 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.012512 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.012521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.012526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.012584 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.017636 (+ 5052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.018638 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.018644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.018648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.018876 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.019517 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.019524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.019531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.019986 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.020212 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.020997 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.021304 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:04:25.023227 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.001993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38524)
0806 09:04:25.002807 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.002831 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.003054 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.003727 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.003732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.004495 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.004937 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.004952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.006562 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.006568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.008377 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.008395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.009792 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.009799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.010007 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.010834 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.010859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.012223 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.017809 (+ 5586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.017816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.017821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.018437 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.019067 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.019073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.019087 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.019359 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.020300 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.020305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.020544 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.022235 (+ 1691us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.022706 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":1779,"threads_started":1}
WARNING: no policy specified for oryx/127.27.187.188@KRBTEST.COM; defaulting to no policy
I20260806 09:04:25.028426 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.002785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.27.187.189:34301)
0806 09:04:25.002998 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.003007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.009164 (+ 6157us) server_negotiation.cc:407] Connection header received
0806 09:04:25.009372 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.009380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.009612 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.009712 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.011776 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.013079 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.015504 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.016473 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.016596 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.020303 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.020312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.020319 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.020360 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.024937 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.025922 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.025928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.025931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.026134 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.026625 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.026630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.026633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.027093 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.027271 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.027918 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.028136 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:04:25.029373 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.003381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56690)
0806 09:04:25.009042 (+ 5661us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.009061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.009249 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.009912 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.009919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.011067 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.011565 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.011585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.013306 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.013314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.015316 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.015337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.017029 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.017040 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.017292 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.018713 (+ 1421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.018756 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.020089 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.025136 (+ 5047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.025145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.025150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.025768 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.026305 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.026312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.026319 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.026504 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.027353 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.027361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.027570 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.028738 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.029061 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5502,"thread_start_us":5288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.188@KRBTEST.COM" created.
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007062, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.190@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
I20260806 09:04:25.061789 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.032284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.27.187.189:34301)
0806 09:04:25.032545 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.032554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.039743 (+ 7189us) server_negotiation.cc:407] Connection header received
0806 09:04:25.040250 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.040257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.040362 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.040499 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.041929 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.043075 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.045289 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.046140 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.046320 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.051772 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.051783 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.051794 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.051846 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.056763 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.057957 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.057964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.057969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.058195 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.059285 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.059292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.059296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.059796 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.059997 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.060735 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.061362 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 09:04:25.061853 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.031755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56692)
0806 09:04:25.039517 (+ 7762us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.039592 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.039939 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.040586 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.040598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.041440 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.041712 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.041725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.043259 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.043265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.044980 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.045003 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.046384 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.046391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.046649 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.047623 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.047652 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.051457 (+ 3805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.057011 (+ 5554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.057033 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.057042 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.057739 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.058625 (+ 886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.058629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.058632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.059065 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.060029 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.060036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.060319 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.060596 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.061330 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7399,"thread_start_us":7197,"threads_started":1}
I20260806 09:04:25.064554 2292 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.187.189:34301: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9"
member_type: NON_VOTER
last_known_addr {
host: "127.27.187.189"
port: 34301
}
attrs {
promote: true
}
}
dest_uuid: "9115f4af30864fbf9fcce0a985db9c4e"
cas_config_opid_index: -1
I20260806 09:04:25.066457 2292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.188.keytab.
Entry for principal oryx/127.27.187.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.188.keytab.
I20260806 09:04:25.072810 2808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:25.073890 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:44171
--webserver_interface=127.27.187.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.188.keytab
--principal=oryx/127.27.187.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:25.071656 2864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.076761 2808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 09:04:25.078200 2863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:25.087023 2864 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.187.189:34301
I20260806 09:04:25.089538 2766 master.cc:537] Master@127.27.187.189:34301 shutting down...
I20260806 09:04:25.090920 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.091333 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:25.119776 2808 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.135159 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.135736 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.136118 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:25.136718 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115f4af30864fbf9fcce0a985db9c4e. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.136999 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:25.135771 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:25.139068 2766 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Raft consensus shutting down.
I20260806 09:04:25.139742 2766 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Raft consensus is shut down!
I20260806 09:04:25.139952 2766 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: stopping tablet replica
I20260806 09:04:25.149642 2766 master.cc:552] Master@127.27.187.189:34301 shutdown complete.
I20260806 09:04:25.149799 2766 master_runner.cc:305] Clearing existing system tablet
I20260806 09:04:25.150283 2766 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:25.150401 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:25.154369 2766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:04:25.156679 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:25.157140 2766 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:25.157677 2766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 09:04:25.157840 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "5f143c3e47bf49fba124d1aef220f7b9"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "89c210c83b4f1cfd7343ec4a649432bb"
server_key_iv: "bbec34287d373fd6801d8e937a7eaad3"
server_key_version: "encryptionkey@0"
I20260806 09:04:25.163142 2766 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:25.170706 2766 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:04:25.171115 2766 log.cc:1196] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 09:04:25.171826 2766 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Deleting consensus metadata
I20260806 09:04:25.172191 2766 master_runner.cc:315] Copying system tablet from 127.27.187.190:38867
I20260806 09:04:25.175568 2766 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: tablet copy: Beginning tablet copy session from remote peer at address 127.27.187.190:38867
I20260806 09:04:25.195114 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.177195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.27.187.190:38867)
0806 09:04:25.177425 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.177433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.177598 (+ 165us) server_negotiation.cc:407] Connection header received
0806 09:04:25.177962 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.177969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.178069 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.178203 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.179432 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.180725 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.182975 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.183665 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.183784 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.186772 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.186781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.186788 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.186839 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.191085 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.192131 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.192138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.192143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.192356 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.192876 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.192882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.192886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.193493 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.193716 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.194387 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.194664 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:04:25.195393 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.176817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38526)
0806 09:04:25.177447 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.177472 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.177757 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.178247 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.178252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.178915 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.179231 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.179250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.180941 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.180947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.182780 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.182798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.184141 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.184148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.184346 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.185220 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.185244 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.186472 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.191313 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.191319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.191324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.191945 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.192513 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.192520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.192524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.192730 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.193767 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.193774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.194048 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.194255 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.194806 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":616,"threads_started":1}
I20260806 09:04:25.196628 2312 tablet_copy_service.cc:140] P 9115f4af30864fbf9fcce0a985db9c4e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5f143c3e47bf49fba124d1aef220f7b9 ({username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:38526)
I20260806 09:04:25.197160 2312 tablet_copy_service.cc:161] P 9115f4af30864fbf9fcce0a985db9c4e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5f143c3e47bf49fba124d1aef220f7b9 at {username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:38526: session id = 5f143c3e47bf49fba124d1aef220f7b9-00000000000000000000000000000000
I20260806 09:04:25.202291 2312 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:04:25.205742 2766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:04:25.214129 2766 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: tablet copy: Starting download of 0 data blocks...
I20260806 09:04:25.214609 2766 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: tablet copy: Starting download of 1 WAL segments...
I20260806 09:04:25.219156 2766 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:04:25.225544 2312 tablet_copy_service.cc:342] P 9115f4af30864fbf9fcce0a985db9c4e: Request end of tablet copy session 5f143c3e47bf49fba124d1aef220f7b9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:38526
I20260806 09:04:25.225886 2312 tablet_copy_service.cc:433] P 9115f4af30864fbf9fcce0a985db9c4e: ending tablet copy session 5f143c3e47bf49fba124d1aef220f7b9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5f143c3e47bf49fba124d1aef220f7b9
I20260806 09:04:25.242138 2766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:25.245616 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:25.248390 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:25.260819 2766 server_base.cc:1034] running on GCE node
W20260806 09:04:25.261209 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:25.261850 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:25.262797 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:25.264108 2766 hybrid_clock.cc:648] HybridClock initialized: now 1786007065264038 us; error 43 us; skew 500 ppm
I20260806 09:04:25.264364 2766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007065, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:25.266875 2766 init.cc:377] Logged in from keytab as oryx/127.27.187.189@KRBTEST.COM (short username oryx)
I20260806 09:04:25.274708 2766 webserver.cc:460] Webserver started at http://127.27.187.189:37377/ using document root <none> and password file <none>
I20260806 09:04:25.275132 2766 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:25.275262 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:25.279088 2766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:25.281445 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:25.282166 2766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 09:04:25.282330 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "5f143c3e47bf49fba124d1aef220f7b9"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "89c210c83b4f1cfd7343ec4a649432bb"
server_key_iv: "bbec34287d373fd6801d8e937a7eaad3"
server_key_version: "encryptionkey@0"
I20260806 09:04:25.282531 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:25.314841 2766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:25.320562 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:25.365669 2760 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
I20260806 09:04:25.376437 2766 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.189:34301
I20260806 09:04:25.376291 2946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.189:34301 every 8 connection(s)
I20260806 09:04:25.379240 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
I20260806 09:04:25.387362 2947 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:04:25.393719 2947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Bootstrap starting.
I20260806 09:04:25.413513 2947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:04:25.414518 2947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Bootstrap complete.
I20260806 09:04:25.416659 2947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } }
I20260806 09:04:25.417243 2947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f143c3e47bf49fba124d1aef220f7b9, State: Initialized, Role: LEARNER
I20260806 09:04:25.417804 2947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } }
I20260806 09:04:25.420203 2947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:25.419967 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.421164 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:25.442659 2961 catalog_manager.cc:1259] Loaded cluster ID: fc6ce6ad38d34352a7088f73bd0a7208
I20260806 09:04:25.442939 2961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: loading cluster ID for follower catalog manager: success
I20260806 09:04:25.449906 2961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: acquiring CA information for follower catalog manager: success
I20260806 09:04:25.453374 2961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:04:25.633502 2866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:25.633924 2866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:25.634166 2866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:25.644748 2866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:25.645020 2866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:25.645287 2866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:25.645483 2866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:25.645728 2866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:25.658128 2866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.188.keytab
--principal=oryx/127.27.187.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.187.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:25.660027 2866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:25.661782 2866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:25.679770 2968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:25.679927 2970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:25.679834 2967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:25.681484 2866 server_base.cc:1034] running on GCE node
I20260806 09:04:25.682463 2866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:25.684280 2866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:25.685601 2866 hybrid_clock.cc:648] HybridClock initialized: now 1786007065685565 us; error 55 us; skew 500 ppm
I20260806 09:04:25.686091 2866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007065, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:25.689924 2866 init.cc:377] Logged in from keytab as oryx/127.27.187.188@KRBTEST.COM (short username oryx)
I20260806 09:04:25.691880 2866 webserver.cc:460] Webserver started at http://127.27.187.188:38869/ using document root <none> and password file <none>
I20260806 09:04:25.692672 2866 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:25.692832 2866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:25.693181 2866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:25.696573 2866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/instance:
uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
format_stamp: "Formatted at 2026-08-06 09:04:25 on dist-test-slave-8tdl"
server_key: "f980e69091e4ba0a618b3398193ed77c"
server_key_iv: "e74aee491ba244df6940df2265ffd765"
server_key_version: "encryptionkey@0"
I20260806 09:04:25.697467 2866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal/instance:
uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
format_stamp: "Formatted at 2026-08-06 09:04:25 on dist-test-slave-8tdl"
server_key: "f980e69091e4ba0a618b3398193ed77c"
server_key_iv: "e74aee491ba244df6940df2265ffd765"
server_key_version: "encryptionkey@0"
I20260806 09:04:25.703749 2866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:04:25.706220 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.706800 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:25.714020 2983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:25.716859 2866 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.001s
I20260806 09:04:25.717123 2866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
format_stamp: "Formatted at 2026-08-06 09:04:25 on dist-test-slave-8tdl"
server_key: "f980e69091e4ba0a618b3398193ed77c"
server_key_iv: "e74aee491ba244df6940df2265ffd765"
server_key_version: "encryptionkey@0"
I20260806 09:04:25.717453 2866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:25.723723 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.706417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56706)
0806 09:04:25.713968 (+ 7551us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.714041 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.714375 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.715010 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.715015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.715086 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:25.715532 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.715546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.717264 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.717271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.719964 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.719991 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.722472 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.722508 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.722929 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.723024 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.723145 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.723313 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7358,"thread_start_us":7186,"threads_started":1}
I20260806 09:04:25.724187 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.706986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.27.187.189:34301)
0806 09:04:25.707856 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.707866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.714203 (+ 6337us) server_negotiation.cc:407] Connection header received
0806 09:04:25.714605 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.714611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.714763 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.714937 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:25.715740 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.717039 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.720231 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.722153 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.722449 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.722745 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.723214 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.723531 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":334,"threads_started":1}
I20260806 09:04:25.726663 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115f4af30864fbf9fcce0a985db9c4e. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: NON_VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: true } } }
I20260806 09:04:25.727039 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:25.728559 2977 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: attempting to promote NON_VOTER 5f143c3e47bf49fba124d1aef220f7b9 to VOTER
I20260806 09:04:25.729573 2977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } }
I20260806 09:04:25.731709 2919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 LEARNER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 09:04:25.732507 2979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 09:04:25.736315 2977 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } }
I20260806 09:04:25.737392 2919 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } }
I20260806 09:04:25.740803 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } }
I20260806 09:04:25.741075 2979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } }
I20260806 09:04:25.741240 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:25.741376 2979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:25.741875 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } }
I20260806 09:04:25.742157 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:25.749877 2866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:25.755455 2866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:25.755762 2866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:25.781636 2866 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.188:44171
I20260806 09:04:25.781699 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.188:44171 every 8 connection(s)
I20260806 09:04:25.783656 2866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
I20260806 09:04:25.787653 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 2866
I20260806 09:04:25.791924 3037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:25.804035 3037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007065, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.188@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:25.826722 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.791375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.27.187.188:44171)
0806 09:04:25.793112 (+ 1737us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.793128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.793210 (+ 82us) server_negotiation.cc:407] Connection header received
0806 09:04:25.795889 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.795918 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.796405 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.796988 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.799354 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.801593 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.803376 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.804214 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.804536 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.813086 (+ 8550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.813099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.813116 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.813203 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.820129 (+ 6926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.821445 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.821455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.821464 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.821786 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.822400 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.822406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.822410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.823591 (+ 1181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.823932 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.824646 (+ 714us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.825256 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":906,"threads_started":1}
I20260806 09:04:25.834085 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.807725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38530)
0806 09:04:25.809226 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.809275 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.809533 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.810926 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.810932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.812294 (+ 1362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.813100 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.813114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.814931 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.814938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.818288 (+ 3350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.818314 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.819569 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.819592 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.819965 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.821172 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.821224 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.825488 (+ 4264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.830262 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.830275 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.830282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.831095 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.831677 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.831683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.831687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.831883 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.832640 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.832649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.832868 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.833113 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.833548 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":874,"threads_started":1}
I20260806 09:04:25.834105 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.807879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.27.187.190:38867)
0806 09:04:25.810375 (+ 2496us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.810381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.810428 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:04:25.810504 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.810510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.810629 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.810797 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.813409 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.814594 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.818578 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.819231 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.819359 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.825781 (+ 6422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.825788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.825792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.825833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.830034 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.831277 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.831283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.831287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.831531 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.832010 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.832015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.832019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.832455 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.832647 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.833281 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.833547 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292,"thread_start_us":2080,"threads_started":1}
I20260806 09:04:25.835987 3037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007065, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.188@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
I20260806 09:04:25.853440 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.836927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56718)
0806 09:04:25.837159 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.837177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.837305 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.837654 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.837660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.838204 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.838409 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.838418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.839724 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.839728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.840949 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.840961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.842093 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.842099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.842352 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.843211 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.843234 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.846159 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.850438 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.850443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.850446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.851107 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.851715 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.851718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.851721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.851902 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.852525 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.852528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.852688 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.852817 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.853120 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 09:04:25.853639 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.837002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.27.187.189:34301)
0806 09:04:25.837240 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.837247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.837270 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:04:25.837430 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.837434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.837499 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.837583 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.838544 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.839530 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.841093 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.841881 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.841988 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.846445 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.846450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.846454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.846494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.850207 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.851269 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.851274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.851277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.851580 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.852013 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.852017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.852019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.852406 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.852529 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.852940 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.853175 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:04:25.854952 3037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007065, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.188@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
I20260806 09:04:25.872303 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.855920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.27.187.188:44171)
0806 09:04:25.856086 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.856093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.856114 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:04:25.856160 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.856164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.856269 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.856380 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.857363 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.858262 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.859855 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.860497 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.860647 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.864784 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.864790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.864793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.864823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.868305 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.869077 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.869082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.869086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.869439 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.870406 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.870412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.870415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.870976 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.871156 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.871621 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.871893 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":71}
I20260806 09:04:25.872382 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.855698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.0.0.1:59630)
0806 09:04:25.855902 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.855919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.856059 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.856394 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.856398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.856992 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.857213 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.857223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.858391 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.858398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.859631 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.859645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.860706 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.860711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.860987 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.861785 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.861802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.864544 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.868471 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.868478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.868482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.868929 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.869581 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.869586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.869590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.870012 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.871094 (+ 1082us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.871098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.871268 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.871492 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.871736 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:04:25.878103 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Bootstrap starting.
I20260806 09:04:25.882577 3037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:25.884500 3037 log.cc:824] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:25.888337 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: No bootstrap required, opened a new log
I20260806 09:04:25.897899 3037 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } }
I20260806 09:04:25.898397 3037 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:25.898571 3037 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4fb41d982b4fbf82dacc12ca19344a, State: Initialized, Role: FOLLOWER
I20260806 09:04:25.899303 3037 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } }
I20260806 09:04:25.900714 3044 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } } }
I20260806 09:04:25.901477 3044 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:25.901744 3037 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:04:25.916143 3056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:04:25.916373 3056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:04:25.919494 2866 master_runner.cc:428] Detected that this master dc4fb41d982b4fbf82dacc12ca19344a is joining an existing cluster
I20260806 09:04:25.919667 2866 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.187.188:44171
I20260806 09:04:25.934489 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2355
I20260806 09:04:25.945214 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.925129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.27.187.190:38867)
0806 09:04:25.925363 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.925369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.926544 (+ 1175us) server_negotiation.cc:407] Connection header received
0806 09:04:25.926820 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.926826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.926927 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.927042 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.928343 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.929103 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.930764 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.931276 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.931433 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.934741 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.934748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.934753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.934801 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.939348 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.941034 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.941045 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.941051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.941416 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.942118 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.942127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.942133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.942698 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.942901 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.944271 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.944642 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:04:25.952018 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.925295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38540)
0806 09:04:25.926375 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.926401 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.926620 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.927076 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.927080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.927805 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.928171 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.928181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.929282 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.929293 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.930582 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.930596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.931496 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.931502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.931695 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.932997 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.933022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.934450 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.939712 (+ 5262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.939736 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.939739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.940762 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.941642 (+ 880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.941651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.941657 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.941937 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.943038 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.943045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.943372 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.943601 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.950336 (+ 6735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":601,"threads_started":1}
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.129@KRBTEST.COM
I20260806 09:04:25.972914 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.925297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.27.187.189:34301)
0806 09:04:25.925539 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.925545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.941404 (+ 15859us) server_negotiation.cc:407] Connection header received
0806 09:04:25.943234 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.943241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.943344 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.943482 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.947669 (+ 4187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.949454 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.958373 (+ 8919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.959526 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.959670 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.964036 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.964046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.964052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.964108 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.968727 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.969921 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.969929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.969933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.970104 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.970903 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.970911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.970915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.971315 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.971461 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.972190 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.972467 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 09:04:25.974586 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.926148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56732)
0806 09:04:25.941253 (+ 15105us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.941292 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.942801 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.943831 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.943836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.946478 (+ 2642us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.947411 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.947434 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.949809 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.949818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.958124 (+ 8306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.958145 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.960277 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.960300 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.960673 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.961804 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.961838 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.963703 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.968929 (+ 5226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.968935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.968939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.969706 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.970286 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.970293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.970300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.970746 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.971575 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.971579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.971807 (+ 228us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.973515 (+ 1708us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.973944 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14925,"mutex_wait_us":61,"thread_start_us":15184,"threads_started":1}
I20260806 09:04:25.977948 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.936596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.27.187.129:46827)
0806 09:04:25.946103 (+ 9507us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.946116 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.946176 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:04:25.946271 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.946284 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.946631 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.946907 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.948990 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.951274 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.955134 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.956953 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.967625 (+ 10672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.967983 (+ 358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.967997 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.968019 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.968056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.972703 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.973769 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.973780 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.973783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.973958 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.974529 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.974536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.974541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.975053 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.977259 (+ 2206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.977408 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.977635 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9295,"thread_start_us":9090,"threads_started":1}
I20260806 09:04:25.979702 2480 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 06 09:04:25 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007062, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.190@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
I20260806 09:04:25.992419 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.942010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.27.187.188:44171)
0806 09:04:25.942229 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.942235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.953758 (+ 11523us) server_negotiation.cc:407] Connection header received
0806 09:04:25.953947 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.953952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.954072 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.954170 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.969455 (+ 15285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.970658 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.974728 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.975685 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.975820 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.983523 (+ 7703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.983531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.983536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.983578 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.988972 (+ 5394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.989959 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.989965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.989968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.990125 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:25.990574 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:25.990579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:25.990582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:25.990987 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:25.991140 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:25.991675 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 09:04:25.991934 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:04:25.993713 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.942356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.0.0.1:59644)
0806 09:04:25.953649 (+ 11293us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.953678 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.953832 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.954258 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.954262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.968440 (+ 14178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.968700 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.968715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.970994 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.970999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.972640 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.972656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.976134 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.976141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.976335 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.977288 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.977311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.978498 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.989155 (+ 10657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.989160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.989165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.989810 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.990268 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.990272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.990276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:25.990455 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:25.991231 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:25.991236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:25.991387 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 09:04:25.992730 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0806 09:04:25.993058 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11082,"thread_start_us":13396,"threads_started":1}
I20260806 09:04:26.004070 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.962414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.0.0.1:59654)
0806 09:04:25.962908 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:25.962938 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:25.963118 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:25.980099 (+ 16981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:25.980107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:25.980731 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:25.981004 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.981022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.984192 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.984198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.986747 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:25.986772 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.988493 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.988501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:25.988773 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.989729 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:25.989759 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:25.992969 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:25.999870 (+ 6901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:25.999877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:25.999883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.000628 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.001447 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.001453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.001457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.001703 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.002737 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:26.002744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:26.002983 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.003177 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.003607 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260806 09:04:26.004292 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:25.964025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.27.187.188:44171)
0806 09:04:25.979625 (+ 15600us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:25.979629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:25.979647 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:04:25.979728 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:25.979731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:25.979830 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:25.981239 (+ 1409us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:25.981755 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.984008 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.986993 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:25.988215 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:25.991688 (+ 3473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:25.993329 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:25.993339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:25.993344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:25.993389 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:25.999650 (+ 6261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.000833 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.000840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.000844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.001043 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.001891 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.001897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.001901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.002497 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.002663 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.003416 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.003685 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15470,"thread_start_us":19190,"threads_started":1}
I20260806 09:04:26.006989 2292 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.187.188:44171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
member_type: NON_VOTER
last_known_addr {
host: "127.27.187.188"
port: 44171
}
attrs {
promote: true
}
}
dest_uuid: "9115f4af30864fbf9fcce0a985db9c4e"
cas_config_opid_index: 6
W20260806 09:04:26.008476 2990 tablet.cc:2369] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:26.008736 2292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } }
I20260806 09:04:26.012342 2919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 09:04:26.013417 2979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 09:04:26.013818 3011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:26.018097 3011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 09:04:26.020751 2979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:26.021453 2978 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.026010 2978 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.187.188:44171
I20260806 09:04:26.027099 2979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.027107 2919 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.027557 2979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:26.027833 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.028120 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:26.031024 2866 master.cc:537] Master@127.27.187.188:44171 shutting down...
I20260806 09:04:26.033130 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.033452 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:04:26.041673 2277 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 09:04:26.048177 2277 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e -> Peer dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171): Couldn't send request to peer dc4fb41d982b4fbf82dacc12ca19344a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 09:04:26.065335 2866 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:26.065824 2866 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:26.066058 2866 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: stopping tablet replica
I20260806 09:04:26.073372 2866 master.cc:552] Master@127.27.187.188:44171 shutdown complete.
I20260806 09:04:26.073490 2866 master_runner.cc:305] Clearing existing system tablet
I20260806 09:04:26.073912 2866 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:26.074033 2866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:26.076728 2866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 09:04:26.078519 3070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:26.079348 2866 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:26.079645 2866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 09:04:26.079779 2866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
format_stamp: "Formatted at 2026-08-06 09:04:25 on dist-test-slave-8tdl"
server_key: "f980e69091e4ba0a618b3398193ed77c"
server_key_iv: "e74aee491ba244df6940df2265ffd765"
server_key_version: "encryptionkey@0"
I20260806 09:04:26.083992 2866 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:26.089710 2866 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:04:26.089936 2866 log.cc:1196] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 09:04:26.090444 2866 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Deleting consensus metadata
I20260806 09:04:26.090835 2866 master_runner.cc:315] Copying system tablet from 127.27.187.190:38867
I20260806 09:04:26.094259 2866 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.187.190:38867
I20260806 09:04:26.110711 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.095270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.27.187.190:38867)
0806 09:04:26.095611 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.095620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.096314 (+ 694us) server_negotiation.cc:407] Connection header received
0806 09:04:26.096593 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.096598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.096682 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.096795 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:26.098107 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.098984 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.100350 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.100794 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.100937 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.103316 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:26.103322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:26.103325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:26.103361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:26.107278 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.108182 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.108189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.108196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.108361 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.108908 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.108917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.108920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.109301 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.109473 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.109932 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.110210 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260806 09:04:26.111024 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.095174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38542)
0806 09:04:26.096173 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:26.096208 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:26.096420 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:26.096823 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:26.096828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:26.097550 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:26.097944 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.097955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.099130 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.099135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.100183 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.100199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.100974 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.100980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.101200 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.101978 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:26.101997 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:26.103063 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:26.107473 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.107479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.107484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.108034 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.108542 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.108550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.108552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.108790 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.109447 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:26.109452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:26.109644 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.109830 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.110229 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":645,"threads_started":1}
I20260806 09:04:26.111873 2312 tablet_copy_service.cc:140] P 9115f4af30864fbf9fcce0a985db9c4e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dc4fb41d982b4fbf82dacc12ca19344a ({username='oryx', principal='oryx/127.27.187.188@KRBTEST.COM'} at 127.0.0.1:38542)
I20260806 09:04:26.112088 2312 tablet_copy_service.cc:161] P 9115f4af30864fbf9fcce0a985db9c4e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dc4fb41d982b4fbf82dacc12ca19344a at {username='oryx', principal='oryx/127.27.187.188@KRBTEST.COM'} at 127.0.0.1:38542: session id = dc4fb41d982b4fbf82dacc12ca19344a-00000000000000000000000000000000
I20260806 09:04:26.114763 2312 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:04:26.116822 2866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:04:26.124660 2866 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: tablet copy: Starting download of 0 data blocks...
I20260806 09:04:26.125154 2866 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: tablet copy: Starting download of 1 WAL segments...
I20260806 09:04:26.126844 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2355
I20260806 09:04:26.132735 2866 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:04:26.137271 2312 tablet_copy_service.cc:342] P 9115f4af30864fbf9fcce0a985db9c4e: Request end of tablet copy session dc4fb41d982b4fbf82dacc12ca19344a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.187.188@KRBTEST.COM'} at 127.0.0.1:38542
I20260806 09:04:26.137508 2312 tablet_copy_service.cc:433] P 9115f4af30864fbf9fcce0a985db9c4e: ending tablet copy session dc4fb41d982b4fbf82dacc12ca19344a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dc4fb41d982b4fbf82dacc12ca19344a
I20260806 09:04:26.149501 2866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:26.152791 3077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:26.153666 3078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:26.154724 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:46827
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:38867
--webserver_port=36809
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:26.158668 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:26.164921 2866 server_base.cc:1034] running on GCE node
I20260806 09:04:26.165508 2866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:26.166292 2866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:26.167515 2866 hybrid_clock.cc:648] HybridClock initialized: now 1786007066167465 us; error 36 us; skew 500 ppm
I20260806 09:04:26.167834 2866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007066, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:26.170503 2866 init.cc:377] Logged in from keytab as oryx/127.27.187.188@KRBTEST.COM (short username oryx)
I20260806 09:04:26.176200 2866 webserver.cc:460] Webserver started at http://127.27.187.188:41697/ using document root <none> and password file <none>
I20260806 09:04:26.176733 2866 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:26.176860 2866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:26.180841 2866 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:26.183275 3088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:26.183830 2866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 09:04:26.184019 2866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
format_stamp: "Formatted at 2026-08-06 09:04:25 on dist-test-slave-8tdl"
server_key: "f980e69091e4ba0a618b3398193ed77c"
server_key_iv: "e74aee491ba244df6940df2265ffd765"
server_key_version: "encryptionkey@0"
I20260806 09:04:26.184267 2866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:26.202798 2866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:26.207533 2866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:26.233429 2866 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.188:44171
I20260806 09:04:26.233487 3139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.188:44171 every 8 connection(s)
I20260806 09:04:26.234786 2866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
I20260806 09:04:26.240360 3140 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:04:26.244714 3140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Bootstrap starting.
I20260806 09:04:26.263275 3140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:04:26.263855 3140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: Bootstrap complete.
I20260806 09:04:26.265033 3140 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } }
I20260806 09:04:26.265379 3140 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc4fb41d982b4fbf82dacc12ca19344a, State: Initialized, Role: LEARNER
I20260806 09:04:26.265712 3140 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } }
I20260806 09:04:26.266629 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.267022 3140 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:26.267191 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: LEARNER
I20260806 09:04:26.277281 3155 catalog_manager.cc:1259] Loaded cluster ID: fc6ce6ad38d34352a7088f73bd0a7208
I20260806 09:04:26.277398 3155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: loading cluster ID for follower catalog manager: success
I20260806 09:04:26.281150 3155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: acquiring CA information for follower catalog manager: success
I20260806 09:04:26.283962 3155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:04:26.502130 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.494036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.0.0.1:59662)
0806 09:04:26.494235 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:26.494264 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:26.494444 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:26.495583 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:26.495588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:26.495709 (+ 121us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:26.496031 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.496043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.497487 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.497491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.499085 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.499101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.500767 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.500775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.501175 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.501203 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.501320 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.501532 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 09:04:26.502173 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.494466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.27.187.188:44171)
0806 09:04:26.495237 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.495242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.495260 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:04:26.495325 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.495329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.495403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.495551 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:26.496216 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.497200 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.499289 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.500497 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.500807 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.501099 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.501361 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.501549 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":468,"threads_started":1}
I20260806 09:04:26.504169 3115 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260806 09:04:26.505079 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115f4af30864fbf9fcce0a985db9c4e. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.505504 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: LEARNER
W20260806 09:04:26.597872 3082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:26.598201 3082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:26.598304 3082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:26.606365 3082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:26.606498 3082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:26.606623 3082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:26.617489 3082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:46827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=36809
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:26.618955 3082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:26.620600 3082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:26.637890 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:26.637989 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:26.637890 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:26.640403 3082 server_base.cc:1034] running on GCE node
I20260806 09:04:26.641458 3082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:26.642884 3082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:26.644219 3082 hybrid_clock.cc:648] HybridClock initialized: now 1786007066644061 us; error 94 us; skew 500 ppm
I20260806 09:04:26.644685 3082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007066, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:26.649546 3082 init.cc:377] Logged in from keytab as oryx/127.27.187.129@KRBTEST.COM (short username oryx)
I20260806 09:04:26.651995 3082 webserver.cc:460] Webserver started at http://127.27.187.129:36809/ using document root <none> and password file <none>
I20260806 09:04:26.653002 3082 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:26.653210 3082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:26.660662 3082 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 09:04:26.664577 3171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:26.666286 3082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:04:26.666564 3082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "3566399ca52e4a689558c05369c24580"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "a232bce24916ecc5061df9de7c60ee0d"
server_key_iv: "4935371ef941ee9859f5244c37ed0606"
server_key_version: "encryptionkey@0"
I20260806 09:04:26.668401 3082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:26.689126 3082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:26.695199 3082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:26.695650 3082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:26.697264 3082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:26.699165 3082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:26.699302 3082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:26.699448 3082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:26.699595 3082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:26.774423 3082 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:46827
I20260806 09:04:26.774601 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:46827 every 8 connection(s)
I20260806 09:04:26.776324 3082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:26.784286 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 3082
I20260806 09:04:26.784734 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2492
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007066, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.130@KRBTEST.COM
I20260806 09:04:26.814117 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.780401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:51233 (local address 127.27.187.189:34301)
0806 09:04:26.780948 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.780956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.782481 (+ 1525us) server_negotiation.cc:407] Connection header received
0806 09:04:26.786593 (+ 4112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.786602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.786951 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.787071 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:26.790192 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.791572 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.794876 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.795589 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.795740 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.801973 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:26.801981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:26.801986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:26.802040 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:26.807600 (+ 5560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.810057 (+ 2457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.810065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.810071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.810329 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.811407 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.811414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.811434 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.811895 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.812109 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.813118 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.813442 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":266,"threads_started":1}
I20260806 09:04:26.816025 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.781068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.27.187.129:51233)
0806 09:04:26.782299 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:26.782363 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:26.786205 (+ 3842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:26.787746 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:26.787762 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:26.788840 (+ 1078us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:26.789907 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.789942 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.791869 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.791877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.794648 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.794700 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.795920 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.795931 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.796347 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.797375 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:26.797434 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:26.801622 (+ 4188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:26.807848 (+ 6226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.807863 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.807906 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.809000 (+ 1094us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.810497 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.810502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.810505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.811220 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.812097 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:26.812106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:26.812652 (+ 546us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.812998 (+ 346us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.813872 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":994,"threads_started":1}
I20260806 09:04:26.818007 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.786377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.27.187.130:35413)
0806 09:04:26.792216 (+ 5839us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.792227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.792267 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:04:26.792334 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.792340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.792563 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.792747 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:26.794550 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.796133 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.797725 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.798395 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.798551 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.804658 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:26.804678 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:26.804703 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:26.804759 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:26.809944 (+ 5185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.811489 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.811498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.811500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.811713 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.812253 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.812260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.812264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.812832 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.813319 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.816772 (+ 3453us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.817191 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5560,"thread_start_us":5370,"threads_started":1}
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007066, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:26.819871 3289 heartbeater.cc:347] Connected to a master server at 127.27.187.189:34301
I20260806 09:04:26.820287 3289 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:26.820570 2618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:26.822002 3289 heartbeater.cc:511] Master 127.27.187.189:34301 requested a full tablet report, sending...
I20260806 09:04:26.825482 2911 ts_manager.cc:194] Registered new tserver with Master: 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827)
Aug 06 09:04:26 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007066, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
I20260806 09:04:26.831103 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.782081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:60905)
0806 09:04:26.786346 (+ 4265us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:26.786368 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:26.786523 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:26.804453 (+ 17930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:26.804458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:26.805082 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:26.808320 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.808340 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.810661 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.810666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.813154 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.813172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.814895 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.814904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.815238 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.816117 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:26.816135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:26.819290 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:26.826444 (+ 7154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.826454 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.826461 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.827483 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.828716 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.828722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.828726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.829006 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.829735 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:26.829740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:26.829903 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.830063 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.830391 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4111,"spinlock_wait_cycles":34560,"thread_start_us":4788,"threads_started":1}
I20260806 09:04:26.831629 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.782019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:60905 (local address 127.27.187.190:38867)
0806 09:04:26.803959 (+ 21940us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.803966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.803989 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:04:26.804059 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.804067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.804177 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.804309 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:26.809184 (+ 4875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.810402 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.813511 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.814567 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.814881 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.819637 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:26.819643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:26.819647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:26.819686 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:26.826216 (+ 6530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.828363 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.828372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.828376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.828559 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.829192 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.829198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.829202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.829620 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.829780 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.830276 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.830774 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21788,"thread_start_us":23368,"threads_started":1}
I20260806 09:04:26.833216 3288 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:26.833472 3288 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:26.834199 3288 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:26.836264 2292 ts_manager.cc:194] Re-registered known tserver with Master: 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827)
I20260806 09:04:26.837765 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:60905
I20260806 09:04:26.840549 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.786337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:38427 (local address 127.27.187.188:44171)
0806 09:04:26.786549 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:26.786555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:26.803400 (+ 16845us) server_negotiation.cc:407] Connection header received
0806 09:04:26.803586 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:26.803595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:26.803691 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:26.803811 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:26.808488 (+ 4677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.809858 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.819985 (+ 10127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.821017 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.821139 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.830461 (+ 9322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:26.830468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:26.830473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:26.830520 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:26.835736 (+ 5216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.836982 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.836988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.836992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.837257 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:26.837851 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:26.837856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:26.837859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:26.838507 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:26.838756 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:26.839916 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0806 09:04:26.840183 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:04:26.841711 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:26.783374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.27.187.129:38427)
0806 09:04:26.803275 (+ 19901us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:26.803297 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:26.803455 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:26.804064 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:26.804073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:26.805113 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:26.808202 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.808227 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.810544 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.810550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.819741 (+ 9191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:26.819759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:26.821783 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:26.821792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:26.822120 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:26.823087 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:26.823111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:26.830200 (+ 7089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:26.835942 (+ 5742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.835948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.835951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.836815 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.837428 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:26.837435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:26.837439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:26.837665 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:26.838926 (+ 1261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:26.838933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:26.839472 (+ 539us) client_negotiation.cc:769] Sending connection context
0806 09:04:26.840824 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0806 09:04:26.841209 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19739,"mutex_wait_us":1,"thread_start_us":19559,"threads_started":1}
I20260806 09:04:26.843154 3287 heartbeater.cc:347] Connected to a master server at 127.27.187.188:44171
I20260806 09:04:26.843395 3287 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:26.844198 3287 heartbeater.cc:511] Master 127.27.187.188:44171 requested a full tablet report, sending...
I20260806 09:04:26.846854 3105 ts_manager.cc:194] Registered new tserver with Master: 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827)
I20260806 09:04:26.927915 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2492
I20260806 09:04:26.946826 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:35413
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:38867
--webserver_port=35287
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:26.984048 2978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: NON_VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: true } } }
I20260806 09:04:26.984543 2978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:26.987028 2978 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: attempting to promote NON_VOTER dc4fb41d982b4fbf82dacc12ca19344a to VOTER
I20260806 09:04:26.987710 2978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:26.989789 2919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:04:26.989907 3115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 LEARNER]: Refusing update from remote peer 9115f4af30864fbf9fcce0a985db9c4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:04:26.990586 3302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:04:26.991199 2978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:04:26.994488 3302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:26.995060 2919 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:26.998932 3301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:26.999313 3301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:26.999555 3302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:26.999814 3302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:27.000131 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:27.000682 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:27.001830 2978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:27.002163 2978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:27.002941 3115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:27.008402 3306 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9115f4af30864fbf9fcce0a985db9c4e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:27.008872 3306 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:04:27.340562 3300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:27.340901 3300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:27.341085 3300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:27.349400 3300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:27.349531 3300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:27.349727 3300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:27.360345 3300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:35413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=35287
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:27.361802 3300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:27.363329 3300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:27.379792 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:27.380031 3314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:27.380393 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:27.381742 3300 server_base.cc:1034] running on GCE node
I20260806 09:04:27.382490 3300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:27.383826 3300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:27.385412 3300 hybrid_clock.cc:648] HybridClock initialized: now 1786007067385346 us; error 58 us; skew 500 ppm
I20260806 09:04:27.385828 3300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:27 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007067, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:27.389474 3300 init.cc:377] Logged in from keytab as oryx/127.27.187.130@KRBTEST.COM (short username oryx)
I20260806 09:04:27.391230 3300 webserver.cc:460] Webserver started at http://127.27.187.130:35287/ using document root <none> and password file <none>
I20260806 09:04:27.391990 3300 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:27.392181 3300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:27.398384 3300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:04:27.401465 3321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:27.403213 3300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:27.403476 3300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "f3a3b54ab59840a6a40662b091b81776"
format_stamp: "Formatted at 2026-08-06 09:04:23 on dist-test-slave-8tdl"
server_key: "81d129408a8b9ac48019b3c1eb8b385b"
server_key_iv: "5be51527b51d8e0d3a6f4842fcd26b18"
server_key_version: "encryptionkey@0"
I20260806 09:04:27.404542 3300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:27.437492 3300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:27.442505 3300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:27.442929 3300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:27.444042 3300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:27.445665 3300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:27.445770 3300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:27.445911 3300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:27.445996 3300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:27.485891 3300 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:35413
I20260806 09:04:27.486116 3436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:35413 every 8 connection(s)
I20260806 09:04:27.487780 3300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:27.498010 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 3300
I20260806 09:04:27.498409 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2630
Aug 06 09:04:27 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007067, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
Aug 06 09:04:27 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007067, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
Aug 06 09:04:27 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007067, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
Aug 06 09:04:27 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.131@KRBTEST.COM
I20260806 09:04:27.543533 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.492843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:35473 (local address 127.27.187.190:38867)
0806 09:04:27.505902 (+ 13059us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:27.505913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:27.505950 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:04:27.506050 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:27.506058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:27.506182 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:27.513427 (+ 7245us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:27.513779 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.515706 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.520714 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.522133 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.522323 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.532453 (+ 10130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:27.532463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:27.532469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:27.532517 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:27.536518 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.537623 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.537630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.537634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.538282 (+ 648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.538898 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.538904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.538908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.539808 (+ 900us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:27.539960 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:27.542777 (+ 2817us) server_negotiation.cc:299] Negotiation successful
0806 09:04:27.543053 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12826,"thread_start_us":13543,"threads_started":1}
I20260806 09:04:27.545040 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.507890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:51215 (local address 127.27.187.189:34301)
0806 09:04:27.509655 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:27.509663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:27.524632 (+ 14969us) server_negotiation.cc:407] Connection header received
0806 09:04:27.525085 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:27.525101 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:27.525306 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:27.525399 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:27.527595 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.528737 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.531315 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.532284 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.532387 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.537105 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:27.537112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:27.537117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:27.537162 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:27.541268 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542171 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.542180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.542186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.542366 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542825 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.542830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.542833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.543231 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:27.543355 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:27.544121 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 09:04:27.544495 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1606,"thread_start_us":1183,"threads_started":1}
I20260806 09:04:27.545073 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.492448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:46193 (local address 127.27.187.188:44171)
0806 09:04:27.521477 (+ 29029us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:27.521487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:27.521515 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:04:27.521597 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:27.521606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:27.521793 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:27.525364 (+ 3571us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:27.525561 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.526423 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.529072 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.530126 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.530260 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.536950 (+ 6690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:27.536959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:27.536965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:27.537016 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:27.540994 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542080 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.542088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.542092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.542253 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542757 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.542765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.542770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.543189 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:27.543347 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:27.544157 (+ 810us) server_negotiation.cc:299] Negotiation successful
0806 09:04:27.544508 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28832,"thread_start_us":28659,"threads_started":1}
I20260806 09:04:27.546118 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.492153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:35473)
0806 09:04:27.493461 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:27.493543 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:27.497991 (+ 4448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:27.509850 (+ 11859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:27.509872 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:27.511458 (+ 1586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:27.513099 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.513143 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.516045 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.516055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.520432 (+ 4377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.520463 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.524612 (+ 4149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.524619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.524887 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.526813 (+ 1926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:27.526862 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:27.532063 (+ 5201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:27.536726 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.536738 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.536762 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.537463 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.538434 (+ 971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.538439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.538445 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.538754 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.540061 (+ 1307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:27.540070 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:27.540608 (+ 538us) client_negotiation.cc:769] Sending connection context
0806 09:04:27.540938 (+ 330us) client_negotiation.cc:240] Negotiation successful
0806 09:04:27.544183 (+ 3245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"spinlock_wait_cycles":23296,"thread_start_us":681,"threads_started":1}
I20260806 09:04:27.546294 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.507907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.27.187.130:51215)
0806 09:04:27.524468 (+ 16561us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:27.524534 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:27.524790 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:27.525712 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:27.525717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:27.526719 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:27.527388 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.527406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.528926 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.528932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.531110 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.531129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.532855 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.532863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.533196 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.533896 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:27.533919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:27.536849 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:27.541419 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.541424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.541427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.541950 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.542492 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.542500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.542665 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.543425 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:27.543428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:27.543832 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 09:04:27.545661 (+ 1829us) client_negotiation.cc:240] Negotiation successful
0806 09:04:27.545932 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16314,"mutex_wait_us":7,"spinlock_wait_cycles":1792,"thread_start_us":16075,"threads_started":1}
I20260806 09:04:27.547374 3443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.492357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.27.187.130:46193)
0806 09:04:27.507408 (+ 15051us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:27.507431 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:27.507576 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:27.522129 (+ 14553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:27.522138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:27.523462 (+ 1324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:27.524276 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.524294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.526590 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.526596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.528775 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:27.528790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.530762 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.530773 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:27.531120 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.532081 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:27.532103 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:27.536649 (+ 4546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:27.541147 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.541151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.541155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.541917 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.542379 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:27.542385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:27.542388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:27.542577 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:27.543417 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:27.543422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:27.543657 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 09:04:27.543849 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 09:04:27.546873 (+ 3024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14800,"mutex_wait_us":2588,"spinlock_wait_cycles":75264,"thread_start_us":14487,"threads_started":1}
I20260806 09:04:27.548385 3439 heartbeater.cc:347] Connected to a master server at 127.27.187.189:34301
I20260806 09:04:27.548557 3437 heartbeater.cc:347] Connected to a master server at 127.27.187.188:44171
I20260806 09:04:27.548609 3438 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:27.548789 3438 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:27.548791 3439 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:27.548799 3437 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:27.549928 3437 heartbeater.cc:511] Master 127.27.187.188:44171 requested a full tablet report, sending...
I20260806 09:04:27.550248 3438 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:27.550446 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:27.499809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.27.187.131:45661)
0806 09:04:27.517805 (+ 17996us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:27.517816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:27.523404 (+ 5588us) server_negotiation.cc:407] Connection header received
0806 09:04:27.523648 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:27.523658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:27.523984 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:27.524217 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:27.527696 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.529414 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.531592 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:27.532874 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:27.533034 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:27.540313 (+ 7279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:27.540335 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:27.540356 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:27.540405 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:27.544976 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.546350 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.546360 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.546365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.546578 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:27.547184 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:27.547191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:27.547194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:27.547711 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:27.549621 (+ 1910us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:27.549787 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 09:04:27.550007 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17668,"thread_start_us":16884,"threads_started":1}
I20260806 09:04:27.549950 3439 heartbeater.cc:511] Master 127.27.187.189:34301 requested a full tablet report, sending...
I20260806 09:04:27.551889 2292 ts_manager.cc:194] Re-registered known tserver with Master: f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:27.551997 3105 ts_manager.cc:194] Registered new tserver with Master: f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:27.552093 2754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:27.553282 2911 ts_manager.cc:194] Registered new tserver with Master: f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:27.553462 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:35473
I20260806 09:04:27.643623 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2630
I20260806 09:04:27.662091 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:45661
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:38867
--webserver_port=37695
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:27.841161 3288 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
W20260806 09:04:28.037086 3451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:28.037420 3451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:28.037528 3451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:28.045500 3451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:28.045712 3451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:28.045871 3451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:28.056582 3451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:45661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=37695
--tserver_master_addrs=127.27.187.190:38867,127.27.187.189:34301,127.27.187.188:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:28.058046 3451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:28.059679 3451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:28.075050 3459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:28.075109 3457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:28.075104 3456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:28.081130 3451 server_base.cc:1034] running on GCE node
I20260806 09:04:28.081969 3451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:28.083563 3451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:28.084836 3451 hybrid_clock.cc:648] HybridClock initialized: now 1786007068084780 us; error 51 us; skew 500 ppm
I20260806 09:04:28.085327 3451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:28 dist-test-slave-8tdl krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007068, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:28.089048 3451 init.cc:377] Logged in from keytab as oryx/127.27.187.131@KRBTEST.COM (short username oryx)
I20260806 09:04:28.090950 3451 webserver.cc:460] Webserver started at http://127.27.187.131:37695/ using document root <none> and password file <none>
I20260806 09:04:28.091720 3451 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:28.091861 3451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:28.098146 3451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 09:04:28.102113 3466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:28.104190 3451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:04:28.104354 3451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b"
format_stamp: "Formatted at 2026-08-06 09:04:24 on dist-test-slave-8tdl"
server_key: "dd3df5bb5d7b8a732c3bbae5558ad403"
server_key_iv: "92823554b57518509ca44a888839fbcb"
server_key_version: "encryptionkey@0"
I20260806 09:04:28.105463 3451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:28.136271 3451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:28.141289 3451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:28.141577 3451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:28.142621 3451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:28.144137 3451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:28.144239 3451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:28.144326 3451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:28.144383 3451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:28.180932 3451 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45661
I20260806 09:04:28.181041 3581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45661 every 8 connection(s)
I20260806 09:04:28.182619 3451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:28.192531 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 3451
Aug 06 09:04:28 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007068, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
Aug 06 09:04:28 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007068, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.189@KRBTEST.COM
I20260806 09:04:28.229676 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.186841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.27.187.131:50683)
0806 09:04:28.188036 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.188105 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.191799 (+ 3694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.202045 (+ 10246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.202053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.202804 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:28.203194 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.203212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.205559 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.205569 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.208006 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.208019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.209800 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.209811 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.210128 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.211479 (+ 1351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:28.211513 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:28.215381 (+ 3868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:28.221596 (+ 6215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.221612 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.221644 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.222623 (+ 979us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.223429 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.223437 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.223444 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.223868 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.224804 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:28.224817 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:28.226037 (+ 1220us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.226498 (+ 461us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.227418 (+ 920us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":240,"thread_start_us":459,"threads_started":1}
Aug 06 09:04:28 dist-test-slave-8tdl krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007068, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.188@KRBTEST.COM
I20260806 09:04:28.232028 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.185882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:50683 (local address 127.27.187.189:34301)
0806 09:04:28.201338 (+ 15456us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.201348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.201382 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:04:28.201468 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.201477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.201607 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.203421 (+ 1814us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.203700 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.205293 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.208156 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.209351 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.213473 (+ 4122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.215668 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.215678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.215684 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.215733 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.221220 (+ 5487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.222857 (+ 1637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.222863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.222867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.223090 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.224077 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.224084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.224088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.224588 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.230970 (+ 6382us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.231153 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.231489 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15242,"thread_start_us":15893,"threads_started":1}
I20260806 09:04:28.238907 3584 heartbeater.cc:347] Connected to a master server at 127.27.187.189:34301
I20260806 09:04:28.239300 3584 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:28.240437 3584 heartbeater.cc:511] Master 127.27.187.189:34301 requested a full tablet report, sending...
I20260806 09:04:28.245211 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.186841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:53769)
0806 09:04:28.192817 (+ 5976us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.192838 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.192984 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.199825 (+ 6841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.199845 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.201127 (+ 1282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:28.202382 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.202424 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.204707 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.204716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.207757 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.207782 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.209433 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.209443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.209894 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.211031 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:28.211073 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:28.215071 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:28.237768 (+ 22697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.237774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.237787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.239024 (+ 1237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.241811 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.241819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.241824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.242155 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.243584 (+ 1429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:28.243592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:28.243848 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.244110 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.244541 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5791,"mutex_wait_us":597,"thread_start_us":5103,"threads_started":1}
I20260806 09:04:28.246457 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.185892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:53769 (local address 127.27.187.190:38867)
0806 09:04:28.187297 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.187302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.192993 (+ 5691us) server_negotiation.cc:407] Connection header received
0806 09:04:28.198926 (+ 5933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.198937 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.199079 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.202699 (+ 3620us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.202939 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.204416 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.208199 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.209084 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.210512 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.215411 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.215418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.215423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.215469 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.236045 (+ 20576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.241336 (+ 5291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.241347 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.241354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.241663 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.242810 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.242816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.242820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.243326 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.243479 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.245678 (+ 2199us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.246055 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":1157,"threads_started":1}
I20260806 09:04:28.248136 3583 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38867
I20260806 09:04:28.248387 3583 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:28.250211 3583 heartbeater.cc:511] Master 127.27.187.190:38867 requested a full tablet report, sending...
I20260806 09:04:28.251775 2911 ts_manager.cc:194] Registered new tserver with Master: 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661)
I20260806 09:04:28.252682 2292 ts_manager.cc:194] Re-registered known tserver with Master: 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661)
I20260806 09:04:28.254135 2292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:53769
I20260806 09:04:28.256261 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.189918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.27.187.131:41027)
0806 09:04:28.213712 (+ 23794us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.213735 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.213898 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.215636 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.215640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.216346 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:28.216823 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.216840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.219806 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.219823 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.222438 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.222457 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.224302 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.224309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.224524 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.225601 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:28.225635 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:28.234072 (+ 8437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:28.251906 (+ 17834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.251912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.251915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.252719 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.253421 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:28.253426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:28.253430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:28.253729 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:28.254513 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:28.254521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:28.254762 (+ 241us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.254949 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.255394 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23608,"thread_start_us":24118,"threads_started":1}
I20260806 09:04:28.257779 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.189202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:41027 (local address 127.27.187.188:44171)
0806 09:04:28.215109 (+ 25907us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.215117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.215139 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:04:28.215212 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.215221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.215311 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.215412 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.217091 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.219516 (+ 2425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.222670 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.223821 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.223978 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.234844 (+ 10866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.234852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.234857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.234910 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.251672 (+ 16762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.252941 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.252946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.252950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.253256 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.253916 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.253923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.253930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.254352 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.256650 (+ 2298us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.256816 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.257138 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25734,"thread_start_us":33970,"threads_started":1}
I20260806 09:04:28.260536 3582 heartbeater.cc:347] Connected to a master server at 127.27.187.188:44171
I20260806 09:04:28.260749 3582 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:28.261587 3582 heartbeater.cc:511] Master 127.27.187.188:44171 requested a full tablet report, sending...
I20260806 09:04:28.263725 3105 ts_manager.cc:194] Registered new tserver with Master: 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661)
I20260806 09:04:28.270344 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.222362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.27.187.189:34301)
0806 09:04:28.232323 (+ 9961us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.232329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.233612 (+ 1283us) server_negotiation.cc:407] Connection header received
0806 09:04:28.233678 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.233715 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.233832 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.233942 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.241882 (+ 7940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.243496 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.246270 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.247283 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.247462 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.258032 (+ 10570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.258039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.258044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.258106 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.263717 (+ 5611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.265108 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.265123 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.265128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.265423 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.265964 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.265985 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.266001 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.266532 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.266870 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.267345 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.269779 (+ 2434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9847,"thread_start_us":14677,"threads_started":1}
I20260806 09:04:28.274483 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.215824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.27.187.190:38867)
0806 09:04:28.225422 (+ 9598us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.225431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.231770 (+ 6339us) server_negotiation.cc:407] Connection header received
0806 09:04:28.232677 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.232683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.232778 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.232899 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.249757 (+ 16858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.251254 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.258005 (+ 6751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.259146 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.259313 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.264607 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.264613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.264617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.264662 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.270361 (+ 5699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.271616 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.271621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.271625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.271965 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.272480 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.272488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.272494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.272953 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.273212 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.273856 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.274084 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9392,"thread_start_us":10038,"threads_started":1}
I20260806 09:04:28.278935 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.241605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.27.187.188:44171)
0806 09:04:28.248441 (+ 6836us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.248450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.250220 (+ 1770us) server_negotiation.cc:407] Connection header received
0806 09:04:28.250646 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.250651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.250829 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.250933 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.258052 (+ 7119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.260893 (+ 2841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.264204 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.265198 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.265276 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.269536 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.269543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.269547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.269585 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.274824 (+ 5239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.275751 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.275757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.275760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.276107 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.276567 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.276584 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.276589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.277117 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.277375 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.277965 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.278298 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6640,"thread_start_us":7339,"threads_started":1}
I20260806 09:04:28.281385 2292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:28.291886 2292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:28.328887 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.314764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.27.187.129:46827)
0806 09:04:28.316699 (+ 1935us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.316711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.316750 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:04:28.316893 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.316906 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.317269 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.317546 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:28.320663 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.321982 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.325363 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.327324 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.327532 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.327748 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.328240 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.328446 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1568,"threads_started":1}
I20260806 09:04:28.328917 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.314373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46827 (local address 127.0.0.1:42820)
0806 09:04:28.315738 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.315769 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.315927 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.317551 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.317555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.317582 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:28.319149 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.319164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.322156 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.322163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.325118 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.325136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.327634 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.327645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.327939 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.328003 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.328130 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.328352 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":1319,"threads_started":1}
I20260806 09:04:28.332393 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.314933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45661 (local address 127.0.0.1:52516)
0806 09:04:28.318756 (+ 3823us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.318798 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.318989 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.322530 (+ 3541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.322543 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.322578 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:28.323301 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.323317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.326063 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.326070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.328540 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.328605 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.331104 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.331115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.331333 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.331361 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.331427 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.331594 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"thread_start_us":3418,"threads_started":1}
I20260806 09:04:28.335631 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.315607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.27.187.131:45661)
0806 09:04:28.319958 (+ 4351us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.319968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.320015 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:04:28.320125 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.320134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.320505 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.323577 (+ 3072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:28.323936 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.325871 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.328851 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.330750 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.332199 (+ 1449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.334851 (+ 2652us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.334931 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.335148 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4170,"thread_start_us":3952,"threads_started":1}
I20260806 09:04:28.340420 3516 tablet_service.cc:1467] Processing CreateTablet for tablet e53164d7bcf84043a5816590ec6871bf (DEFAULT_TABLE table=test-table [id=c8577da975f44142afab797f556b3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:28.340778 3219 tablet_service.cc:1467] Processing CreateTablet for tablet e53164d7bcf84043a5816590ec6871bf (DEFAULT_TABLE table=test-table [id=c8577da975f44142afab797f556b3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:28.341923 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53164d7bcf84043a5816590ec6871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:28.342375 3219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53164d7bcf84043a5816590ec6871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:28.345525 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.316779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.27.187.130:35413)
0806 09:04:28.317309 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.317316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.329547 (+ 12231us) server_negotiation.cc:407] Connection header received
0806 09:04:28.330590 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.330606 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.331405 (+ 799us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.331798 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:28.332830 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.334911 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.342439 (+ 7528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.344190 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.344428 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.344648 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.344964 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.345151 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":287,"threads_started":1}
I20260806 09:04:28.345604 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.316110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:35413 (local address 127.0.0.1:53648)
0806 09:04:28.329369 (+ 13259us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.329392 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.329885 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.331788 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.331794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.331839 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:28.332107 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.332131 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.335186 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.335192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.342043 (+ 6851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.342071 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.344386 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.344394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.344636 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.344672 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.344780 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.344946 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13152,"thread_start_us":19494,"threads_started":1}
I20260806 09:04:28.351002 3371 tablet_service.cc:1467] Processing CreateTablet for tablet e53164d7bcf84043a5816590ec6871bf (DEFAULT_TABLE table=test-table [id=c8577da975f44142afab797f556b3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:28.352841 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53164d7bcf84043a5816590ec6871bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:28.359683 3619 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Bootstrap starting.
I20260806 09:04:28.361900 3620 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Bootstrap starting.
I20260806 09:04:28.364732 3619 tablet_bootstrap.cc:654] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:28.366067 3620 tablet_bootstrap.cc:654] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:28.366474 3619 log.cc:824] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:28.367908 3620 log.cc:824] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:28.370757 3619 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: No bootstrap required, opened a new log
I20260806 09:04:28.371373 3619 ts_tablet_manager.cc:1397] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260806 09:04:28.379086 3620 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: No bootstrap required, opened a new log
I20260806 09:04:28.379514 3620 ts_tablet_manager.cc:1397] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20260806 09:04:28.384902 3619 raft_consensus.cc:348] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.385635 3619 raft_consensus.cc:374] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:28.385818 3619 raft_consensus.cc:729] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3566399ca52e4a689558c05369c24580, State: Initialized, Role: FOLLOWER
I20260806 09:04:28.386802 3623 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Bootstrap starting.
I20260806 09:04:28.386715 3619 consensus_queue.cc:260] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.389173 3619 ts_tablet_manager.cc:1428] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20260806 09:04:28.395001 3623 tablet_bootstrap.cc:654] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:28.393946 3620 raft_consensus.cc:348] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.395700 3620 raft_consensus.cc:374] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:28.395823 3620 raft_consensus.cc:729] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463fd3157c8e4c57bbbe56e9ec86cb4b, State: Initialized, Role: FOLLOWER
I20260806 09:04:28.396690 3623 log.cc:824] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:28.396561 3620 consensus_queue.cc:260] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.396903 3624 raft_consensus.cc:482] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:28.397269 3624 raft_consensus.cc:504] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.399114 3620 ts_tablet_manager.cc:1428] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260806 09:04:28.399529 3583 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
I20260806 09:04:28.406380 3624 leader_election.cc:288] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661), f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:28.417403 3623 tablet_bootstrap.cc:492] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: No bootstrap required, opened a new log
I20260806 09:04:28.418011 3623 ts_tablet_manager.cc:1397] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.011s
I20260806 09:04:28.422180 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.400128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:44659 (local address 127.27.187.131:45661)
0806 09:04:28.400509 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.400516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.411699 (+ 11183us) server_negotiation.cc:407] Connection header received
0806 09:04:28.411991 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.412010 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.412176 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.412366 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:28.413894 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.415480 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.419086 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.420606 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.420935 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.421101 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.421638 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.421789 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260806 09:04:28.422435 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.399727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45661 (local address 127.27.187.129:44659)
0806 09:04:28.411565 (+ 11838us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.411586 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.411832 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.412696 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.412703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.412763 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:28.413666 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.413684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.415803 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.415808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.418843 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.418890 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.421133 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.421145 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.421421 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.421458 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.421614 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.421778 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11686,"thread_start_us":11273,"threads_started":1}
I20260806 09:04:28.423691 3536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53164d7bcf84043a5816590ec6871bf" candidate_uuid: "3566399ca52e4a689558c05369c24580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" is_pre_election: true
I20260806 09:04:28.424482 3536 raft_consensus.cc:2454] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3566399ca52e4a689558c05369c24580 in term 0.
I20260806 09:04:28.425575 3174 leader_election.cc:302] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3566399ca52e4a689558c05369c24580, 463fd3157c8e4c57bbbe56e9ec86cb4b; no voters:
I20260806 09:04:28.426276 3624 raft_consensus.cc:2781] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:28.426637 3624 raft_consensus.cc:482] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:28.426925 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.407264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:42119 (local address 127.27.187.130:35413)
0806 09:04:28.407534 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.407542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.414307 (+ 6765us) server_negotiation.cc:407] Connection header received
0806 09:04:28.414726 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.414732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.414815 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.414887 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:28.416743 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.418315 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.422614 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.424709 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.424976 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.425071 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.426326 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.426570 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:04:28.426967 3624 raft_consensus.cc:3037] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:28.427115 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.408075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:35413 (local address 127.27.187.129:42119)
0806 09:04:28.413572 (+ 5497us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:28.414193 (+ 621us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:28.414469 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:28.415133 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:28.415138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:28.415165 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:28.416449 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.416472 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.418635 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.418642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.422311 (+ 3669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:28.422332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.425563 (+ 3231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.425582 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:28.426054 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.426108 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 09:04:28.426250 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 09:04:28.426472 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5190,"thread_start_us":4441,"threads_started":1}
I20260806 09:04:28.430070 3624 raft_consensus.cc:504] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.432173 3624 leader_election.cc:288] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [CANDIDATE]: Term 1 election: Requested vote from peers 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661), f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:28.432024 3536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53164d7bcf84043a5816590ec6871bf" candidate_uuid: "3566399ca52e4a689558c05369c24580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b"
I20260806 09:04:28.432477 3536 raft_consensus.cc:3037] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:28.432276 3391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53164d7bcf84043a5816590ec6871bf" candidate_uuid: "3566399ca52e4a689558c05369c24580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a3b54ab59840a6a40662b091b81776"
I20260806 09:04:28.431695 3392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53164d7bcf84043a5816590ec6871bf" candidate_uuid: "3566399ca52e4a689558c05369c24580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a3b54ab59840a6a40662b091b81776" is_pre_election: true
I20260806 09:04:28.434913 3623 raft_consensus.cc:348] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.435586 3623 raft_consensus.cc:374] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 09:04:28.435519 3585 tablet.cc:2369] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:04:28.435566 3173 leader_election.cc:341] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413): Illegal state: must be running to vote when last-logged opid is not known
I20260806 09:04:28.435770 3623 raft_consensus.cc:729] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a3b54ab59840a6a40662b091b81776, State: Initialized, Role: FOLLOWER
I20260806 09:04:28.436930 3536 raft_consensus.cc:2454] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3566399ca52e4a689558c05369c24580 in term 1.
I20260806 09:04:28.436862 3623 consensus_queue.cc:260] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.437704 3174 leader_election.cc:302] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3566399ca52e4a689558c05369c24580, 463fd3157c8e4c57bbbe56e9ec86cb4b; no voters: f3a3b54ab59840a6a40662b091b81776
I20260806 09:04:28.438215 3624 raft_consensus.cc:2781] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:28.439522 3624 raft_consensus.cc:686] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 1 LEADER]: Becoming Leader. State: Replica: 3566399ca52e4a689558c05369c24580, State: Running, Role: LEADER
I20260806 09:04:28.439826 3623 ts_tablet_manager.cc:1428] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260806 09:04:28.440253 3624 consensus_queue.cc:237] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:28.441717 3438 heartbeater.cc:503] Master 127.27.187.190:38867 was elected leader, sending a full tablet report...
I20260806 09:04:28.447788 2291 catalog_manager.cc:5495] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3566399ca52e4a689558c05369c24580 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "3566399ca52e4a689558c05369c24580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } health_report { overall_health: UNKNOWN } } }
I20260806 09:04:28.492636 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.27.187.129:46827)
0806 09:04:28.485191 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.485198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.486021 (+ 823us) server_negotiation.cc:407] Connection header received
0806 09:04:28.486300 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.486305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.486392 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.486603 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:28.487013 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.487826 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.489705 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.490444 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.490611 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.491323 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:28.491849 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:28.492084 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.492138 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.492233 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 09:04:28.493352 3440 tablet.cc:2369] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:28.504721 3391 raft_consensus.cc:3037] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:28.506212 3536 raft_consensus.cc:1261] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Refusing update from remote peer 3566399ca52e4a689558c05369c24580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:28.507565 3630 consensus_queue.cc:1035] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:28.508355 3391 raft_consensus.cc:1261] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Refusing update from remote peer 3566399ca52e4a689558c05369c24580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:28.509516 3630 consensus_queue.cc:1035] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:28.529120 3634 mvcc.cc:205] Tried to move back new op lower bound from 7315484952579989504 to 7315484952333672448. Current Snapshot: MvccSnapshot[applied={T|T < 7315484952579989504}]
I20260806 09:04:28.535367 3635 mvcc.cc:205] Tried to move back new op lower bound from 7315484952579989504 to 7315484952333672448. Current Snapshot: MvccSnapshot[applied={T|T < 7315484952579989504}]
I20260806 09:04:28.546334 3636 mvcc.cc:205] Tried to move back new op lower bound from 7315484952579989504 to 7315484952333672448. Current Snapshot: MvccSnapshot[applied={T|T < 7315484952579989504}]
I20260806 09:04:28.561210 2292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:28.561494 2292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:28.569342 2292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9115f4af30864fbf9fcce0a985db9c4e: Sending DeleteTablet for 3 replicas of tablet e53164d7bcf84043a5816590ec6871bf
I20260806 09:04:28.570946 3219 tablet_service.cc:1514] Processing DeleteTablet for tablet e53164d7bcf84043a5816590ec6871bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:28 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:42820
I20260806 09:04:28.571197 3371 tablet_service.cc:1514] Processing DeleteTablet for tablet e53164d7bcf84043a5816590ec6871bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:28 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53648
I20260806 09:04:28.572237 3516 tablet_service.cc:1514] Processing DeleteTablet for tablet e53164d7bcf84043a5816590ec6871bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:28 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:52516
I20260806 09:04:28.572458 3645 tablet_replica.cc:330] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: stopping tablet replica
I20260806 09:04:28.573835 3646 tablet_replica.cc:330] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: stopping tablet replica
I20260806 09:04:28.574525 3646 raft_consensus.cc:2229] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:28.575353 3646 raft_consensus.cc:2258] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:28.575896 3645 raft_consensus.cc:2229] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:28.576642 3645 raft_consensus.cc:2258] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:28.579495 3645 ts_tablet_manager.cc:1905] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:28.579914 3646 ts_tablet_manager.cc:1905] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:28.589409 3646 ts_tablet_manager.cc:1918] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:28.589811 3646 log.cc:1196] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/e53164d7bcf84043a5816590ec6871bf
I20260806 09:04:28.590209 3645 ts_tablet_manager.cc:1918] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:28.590410 3645 log.cc:1196] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/e53164d7bcf84043a5816590ec6871bf
I20260806 09:04:28.590790 3646 ts_tablet_manager.cc:1939] T e53164d7bcf84043a5816590ec6871bf P 3566399ca52e4a689558c05369c24580: Deleting consensus metadata
I20260806 09:04:28.591082 3645 ts_tablet_manager.cc:1939] T e53164d7bcf84043a5816590ec6871bf P f3a3b54ab59840a6a40662b091b81776: Deleting consensus metadata
I20260806 09:04:28.593001 2276 catalog_manager.cc:4841] TS 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827): tablet e53164d7bcf84043a5816590ec6871bf (table test-table [id=c8577da975f44142afab797f556b3447]) successfully deleted
I20260806 09:04:28.593411 2276 catalog_manager.cc:4841] TS f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413): tablet e53164d7bcf84043a5816590ec6871bf (table test-table [id=c8577da975f44142afab797f556b3447]) successfully deleted
I20260806 09:04:28.596405 3647 tablet_replica.cc:330] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: stopping tablet replica
I20260806 09:04:28.596853 3647 raft_consensus.cc:2229] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:28.597096 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2259
I20260806 09:04:28.597497 3647 raft_consensus.cc:2258] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:28.598510 2322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:28.602494 3647 ts_tablet_manager.cc:1905] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:28.611554 3647 ts_tablet_manager.cc:1918] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:28.611802 3647 log.cc:1196] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/e53164d7bcf84043a5816590ec6871bf
I20260806 09:04:28.612428 3647 ts_tablet_manager.cc:1939] T e53164d7bcf84043a5816590ec6871bf P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting consensus metadata
I20260806 09:04:28.626333 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.575650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.27.187.189:34301)
0806 09:04:28.575919 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.575926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.595811 (+ 19885us) server_negotiation.cc:407] Connection header received
0806 09:04:28.595902 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.595908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.596001 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.596121 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.601353 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.602828 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.611948 (+ 9120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.613091 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.613276 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.617584 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.617594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.617600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.617673 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.622317 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.623542 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.623551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.623556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.623768 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.624368 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.624374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.624378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.624832 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.625128 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.625598 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.625921 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:04:28.647557 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.595697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.27.187.188:44171)
0806 09:04:28.596404 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.596410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.627661 (+ 31251us) server_negotiation.cc:407] Connection header received
0806 09:04:28.627768 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.627774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.627872 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.627987 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.629932 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.631249 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.633558 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.634442 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.634666 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.639029 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.639038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.639044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.639102 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.643574 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.644578 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.644585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.644590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.644848 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.645374 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.645380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.645384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.645881 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.646315 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.646640 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.647034 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567}
I20260806 09:04:28.743337 2277 catalog_manager.cc:4841] TS 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661): tablet e53164d7bcf84043a5816590ec6871bf (table test-table [id=c8577da975f44142afab797f556b3447]) successfully deleted
I20260806 09:04:28.744118 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2259
I20260806 09:04:28.813112 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.785862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.27.187.189:34301)
0806 09:04:28.786118 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.786126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.787515 (+ 1389us) server_negotiation.cc:407] Connection header received
0806 09:04:28.788048 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.788055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.788151 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.788232 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.791442 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.792712 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.798892 (+ 6180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.799928 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.800106 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.804308 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.804316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.804322 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.804378 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.809053 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.810249 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.810257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.810260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.810465 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.811097 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.811104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.811108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.811593 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.811905 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.812379 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.812722 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:04:28.819973 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:28.787491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.27.187.188:44171)
0806 09:04:28.787707 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:28.787714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:28.793958 (+ 6244us) server_negotiation.cc:407] Connection header received
0806 09:04:28.794197 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:28.794204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:28.794323 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:28.794428 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:28.804256 (+ 9828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.805466 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.807721 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:28.808580 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:28.808863 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:28.812706 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:28.812713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:28.812718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:28.812777 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:28.816604 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.817511 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.817516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.817519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.817677 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:28.818275 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:28.818280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:28.818282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:28.818624 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:28.818980 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:28.819379 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 09:04:28.819618 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:04:29.542974 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:29.541952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:50471)
0806 09:04:29.542535 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:29.542664 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":355,"threads_started":1}
W20260806 09:04:29.543499 3438 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:38867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:29.550133 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:29.549156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:46031)
0806 09:04:29.549644 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:29.549829 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
W20260806 09:04:29.550621 3288 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:38867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:29.562078 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:29.561244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:43033)
0806 09:04:29.561734 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:29.561871 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
W20260806 09:04:29.562614 3583 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:38867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:30.070487 3661 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9115f4af30864fbf9fcce0a985db9c4e)
I20260806 09:04:30.070644 3662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9115f4af30864fbf9fcce0a985db9c4e)
I20260806 09:04:30.071000 3662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:30.070967 3661 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:30.072925 3662 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:30.072964 3661 leader_election.cc:288] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189:34301)
I20260806 09:04:30.084893 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.073057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38960)
0806 09:04:30.083896 (+ 10839us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.084475 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10651,"threads_started":1}
W20260806 09:04:30.087841 3089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:30.088581 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.073021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38944)
0806 09:04:30.087764 (+ 14743us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.088097 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14623,"thread_start_us":14451,"threads_started":1}
W20260806 09:04:30.090016 2895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:30.094357 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.093174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38972)
0806 09:04:30.093666 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.093923 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260806 09:04:30.094374 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.093727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38974)
0806 09:04:30.093938 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.094127 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 09:04:30.094871 3089 leader_election.cc:334] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
W20260806 09:04:30.094888 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:30.098816 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.083452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.27.187.188:44171)
0806 09:04:30.087200 (+ 3748us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:30.087209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:30.089492 (+ 2283us) server_negotiation.cc:407] Connection header received
0806 09:04:30.089616 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:30.089625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:30.089754 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:30.089858 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:30.091351 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.093878 (+ 2527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.096236 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.097520 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.097719 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:30.097805 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:30.098245 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 09:04:30.098410 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":680,"server-negotiator.queue_time_us":3586,"thread_start_us":3375,"threads_started":1}
I20260806 09:04:30.098887 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.073819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:44171 (local address 127.0.0.1:59682)
0806 09:04:30.089010 (+ 15191us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.089039 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:30.089292 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:30.090215 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:30.090222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:30.090258 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:30.090577 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:30.090598 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.094205 (+ 3607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.094216 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:30.096047 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:30.096063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.097788 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.097794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:30.098025 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:30.098060 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:04:30.098153 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 09:04:30.098365 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15008,"mutex_wait_us":28,"thread_start_us":15486,"threads_started":1}
I20260806 09:04:30.099387 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.074976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:34301 (local address 127.0.0.1:56746)
0806 09:04:30.086130 (+ 11154us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.086175 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:30.086432 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:30.090732 (+ 4300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:30.090741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:30.090792 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:30.091455 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:30.091482 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.095076 (+ 3594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.095082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:30.096717 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:30.096733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.098445 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.098452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:30.098740 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:30.098785 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 09:04:30.098912 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:04:30.099083 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10981,"mutex_wait_us":62,"thread_start_us":7903,"threads_started":1}
I20260806 09:04:30.099478 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.087832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.27.187.189:34301)
0806 09:04:30.090079 (+ 2247us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:30.090089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:30.090117 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:04:30.090210 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:30.090230 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:30.090360 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:30.091795 (+ 1435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:30.093223 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.094824 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.096874 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:30.098109 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:30.098328 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:30.098401 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:30.098954 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 09:04:30.099115 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2055,"thread_start_us":1851,"threads_started":1}
I20260806 09:04:30.100127 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" is_pre_election: true
I20260806 09:04:30.100136 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f143c3e47bf49fba124d1aef220f7b9" is_pre_election: true
I20260806 09:04:30.100451 2919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4fb41d982b4fbf82dacc12ca19344a in term 1.
I20260806 09:04:30.100497 3115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f143c3e47bf49fba124d1aef220f7b9 in term 1.
I20260806 09:04:30.101092 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters: 9115f4af30864fbf9fcce0a985db9c4e
I20260806 09:04:30.101090 3091 leader_election.cc:302] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters: 9115f4af30864fbf9fcce0a985db9c4e
I20260806 09:04:30.101505 3662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 09:04:30.101529 3661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 09:04:30.101671 3661 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9115f4af30864fbf9fcce0a985db9c4e)
I20260806 09:04:30.101647 3662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9115f4af30864fbf9fcce0a985db9c4e)
I20260806 09:04:30.101783 3661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:04:30.101780 3662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:04:30.104341 3661 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:30.104420 3662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:30.105159 3661 leader_election.cc:288] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 election: Requested vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189:34301)
I20260806 09:04:30.105243 3662 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 election: Requested vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:30.105841 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.105348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:38988)
0806 09:04:30.105526 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.105659 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 09:04:30.105892 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
I20260806 09:04:30.106130 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.105630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39000)
0806 09:04:30.105833 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.105947 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:04:30.106335 3115 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f143c3e47bf49fba124d1aef220f7b9 in current term 2: Already voted for candidate dc4fb41d982b4fbf82dacc12ca19344a in this term.
I20260806 09:04:30.106796 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f143c3e47bf49fba124d1aef220f7b9"
I20260806 09:04:30.107019 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.106614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39002)
0806 09:04:30.106799 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.106883 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:04:30.107121 2919 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dc4fb41d982b4fbf82dacc12ca19344a in current term 2: Already voted for candidate 5f143c3e47bf49fba124d1aef220f7b9 in this term.
I20260806 09:04:30.107259 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.106866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39014)
0806 09:04:30.107034 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.107132 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 09:04:30.107287 3089 leader_election.cc:334] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
W20260806 09:04:30.107682 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:30.107792 3091 leader_election.cc:302] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc4fb41d982b4fbf82dacc12ca19344a; no voters: 5f143c3e47bf49fba124d1aef220f7b9, 9115f4af30864fbf9fcce0a985db9c4e
I20260806 09:04:30.107878 2895 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9; no voters: 9115f4af30864fbf9fcce0a985db9c4e, dc4fb41d982b4fbf82dacc12ca19344a
I20260806 09:04:30.108198 3662 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 09:04:30.108255 3661 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 09:04:30.545337 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.544425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:55703)
0806 09:04:30.544994 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.545128 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":341,"threads_started":1}
I20260806 09:04:30.552646 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.551761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:57451)
0806 09:04:30.552352 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.552449 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":313,"threads_started":1}
I20260806 09:04:30.564157 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:30.563412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:43145)
0806 09:04:30.563892 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:30.563992 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":235,"threads_started":1}
I20260806 09:04:31.547544 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.546480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:33229)
0806 09:04:31.547105 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.547219 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":348,"threads_started":1}
I20260806 09:04:31.555115 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.553987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:48327)
0806 09:04:31.554723 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.554855 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":437,"threads_started":1}
I20260806 09:04:31.565932 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.565177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:44129)
0806 09:04:31.565659 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.565751 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260806 09:04:31.787739 3677 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:31.787940 3677 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:31.788704 3677 leader_election.cc:288] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189:34301)
I20260806 09:04:31.788712 3678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:31.789016 3678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:31.789451 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f143c3e47bf49fba124d1aef220f7b9" is_pre_election: true
I20260806 09:04:31.789770 2919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4fb41d982b4fbf82dacc12ca19344a in term 2.
I20260806 09:04:31.789927 3678 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:31.790067 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.788892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39266)
0806 09:04:31.789677 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.789829 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":483,"threads_started":1}
I20260806 09:04:31.790750 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" is_pre_election: true
I20260806 09:04:31.790920 3091 leader_election.cc:302] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters:
I20260806 09:04:31.791105 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.790004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39276)
0806 09:04:31.790741 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.790900 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":395,"threads_started":1}
I20260806 09:04:31.791101 3115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f143c3e47bf49fba124d1aef220f7b9 in term 2.
I20260806 09:04:31.791286 3677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 09:04:31.791416 3677 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:31.791527 3677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:04:31.791957 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters:
I20260806 09:04:31.792357 3678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 09:04:31.792492 3678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:31.792601 3678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:04:31.796190 3677 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:31.798369 3677 leader_election.cc:288] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 election: Requested vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), 5f143c3e47bf49fba124d1aef220f7b9 (127.27.187.189:34301)
I20260806 09:04:31.800043 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.799482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39298)
0806 09:04:31.799746 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.799837 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 09:04:31.800376 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.799942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39308)
0806 09:04:31.800087 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.800200 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 09:04:31.800416 3089 leader_election.cc:334] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:31.800465 2919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f143c3e47bf49fba124d1aef220f7b9"
W20260806 09:04:31.800640 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:31.797211 3678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:31.801060 2919 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dc4fb41d982b4fbf82dacc12ca19344a in current term 3: Already voted for candidate 5f143c3e47bf49fba124d1aef220f7b9 in this term.
I20260806 09:04:31.801833 3678 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 election: Requested vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:31.801891 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.801309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39318)
0806 09:04:31.801611 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.801718 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 09:04:31.802412 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.801925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39322)
0806 09:04:31.802078 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.802185 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 09:04:31.802412 3089 leader_election.cc:334] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:31.802547 3089 leader_election.cc:302] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc4fb41d982b4fbf82dacc12ca19344a; no voters: 5f143c3e47bf49fba124d1aef220f7b9, 9115f4af30864fbf9fcce0a985db9c4e
I20260806 09:04:31.802490 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
I20260806 09:04:31.802906 3115 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f143c3e47bf49fba124d1aef220f7b9 in current term 3: Already voted for candidate dc4fb41d982b4fbf82dacc12ca19344a in this term.
I20260806 09:04:31.803097 3677 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 09:04:31.803594 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:31.803188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:39336)
0806 09:04:31.803382 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:31.803480 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 09:04:31.803754 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:31.803895 2895 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9; no voters: 9115f4af30864fbf9fcce0a985db9c4e, dc4fb41d982b4fbf82dacc12ca19344a
I20260806 09:04:31.804167 3678 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 09:04:32.549582 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:32.548703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:47377)
0806 09:04:32.549272 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:32.549374 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":311,"threads_started":1}
I20260806 09:04:32.557338 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:32.556261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:40703)
0806 09:04:32.556972 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:32.557104 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":468,"threads_started":1}
I20260806 09:04:32.567894 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:32.567027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:55621)
0806 09:04:32.567599 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:32.567719 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":386,"threads_started":1}
I20260806 09:04:33.551411 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:33.550603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:58593)
0806 09:04:33.551085 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:33.551172 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":282,"threads_started":1}
I20260806 09:04:33.559651 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:33.558526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:52621)
0806 09:04:33.559123 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:33.559298 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":359,"threads_started":1}
I20260806 09:04:33.570140 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:33.568846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:39401)
0806 09:04:33.569837 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:33.569952 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":707,"threads_started":1}
I20260806 09:04:34.554158 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.552698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:39187)
0806 09:04:34.553739 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.553873 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":690,"threads_started":1}
I20260806 09:04:34.559020 3690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:34.559309 3690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:34.560029 3690 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:34.560696 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" is_pre_election: true
I20260806 09:04:34.561081 3115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f143c3e47bf49fba124d1aef220f7b9 in term 3.
I20260806 09:04:34.561607 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters:
I20260806 09:04:34.562181 3690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 09:04:34.562418 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.560386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37598)
0806 09:04:34.561371 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.562021 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":307,"threads_started":1}
I20260806 09:04:34.562454 3690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:34.562609 3690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 3 FOLLOWER]: Advancing to term 4
I20260806 09:04:34.562973 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.560847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:60887)
0806 09:04:34.561804 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.561933 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":728,"threads_started":1}
I20260806 09:04:34.564096 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.563652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37600)
0806 09:04:34.563826 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.563937 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 09:04:34.564316 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:34.565637 3690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:34.566315 3690 leader_election.cc:288] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 election: Requested vote from peers 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867), dc4fb41d982b4fbf82dacc12ca19344a (127.27.187.188:44171)
I20260806 09:04:34.566876 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.566339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37610)
0806 09:04:34.566536 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.566622 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:04:34.566917 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f143c3e47bf49fba124d1aef220f7b9" candidate_term: 4 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4fb41d982b4fbf82dacc12ca19344a"
I20260806 09:04:34.567238 3115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 3 FOLLOWER]: Advancing to term 4
I20260806 09:04:34.567972 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.567626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37626)
0806 09:04:34.567782 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.567863 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 09:04:34.568295 2895 leader_election.cc:334] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
I20260806 09:04:34.569900 3115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f143c3e47bf49fba124d1aef220f7b9 in term 4.
I20260806 09:04:34.570611 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f143c3e47bf49fba124d1aef220f7b9, dc4fb41d982b4fbf82dacc12ca19344a; no voters: 9115f4af30864fbf9fcce0a985db9c4e
I20260806 09:04:34.571246 3690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 09:04:34.572188 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.571248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:57745)
0806 09:04:34.571840 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.571955 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":287,"threads_started":1}
I20260806 09:04:34.572779 3690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [term 4 LEADER]: Becoming Leader. State: Replica: 5f143c3e47bf49fba124d1aef220f7b9, State: Running, Role: LEADER
I20260806 09:04:34.573370 3690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } }
I20260806 09:04:34.575331 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f143c3e47bf49fba124d1aef220f7b9. Latest consensus state: current_term: 4 leader_uuid: "5f143c3e47bf49fba124d1aef220f7b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:34.575868 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:34.576704 3697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:34.581192 3697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:34.582141 3697 catalog_manager.cc:1259] Loaded cluster ID: fc6ce6ad38d34352a7088f73bd0a7208
I20260806 09:04:34.582233 3697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:34.583580 3697 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:34.584543 3697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Loaded TSK: 0
I20260806 09:04:34.585436 3697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:04:34.689090 2911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:34.696882 2911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:34.701792 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.701236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37644)
0806 09:04:34.701433 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.701559 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:04:34.701833 3115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [term 4 FOLLOWER]: Refusing update from remote peer 5f143c3e47bf49fba124d1aef220f7b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20260806 09:04:34.703037 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.702502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37658)
0806 09:04:34.702727 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.702846 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 09:04:34.702837 3695 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 09:04:34.703346 2895 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 -> Peer 9115f4af30864fbf9fcce0a985db9c4e (127.27.187.190:38867): Couldn't send request to peer 9115f4af30864fbf9fcce0a985db9c4e. Status: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:04:34.706641 3699 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f143c3e47bf49fba124d1aef220f7b9. Latest consensus state: current_term: 4 leader_uuid: "5f143c3e47bf49fba124d1aef220f7b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:34.707158 3699 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:34.709448 3699 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5f143c3e47bf49fba124d1aef220f7b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:34.709920 3699 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4fb41d982b4fbf82dacc12ca19344a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:04:34.715866 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5f143c3e47bf49fba124d1aef220f7b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9115f4af30864fbf9fcce0a985db9c4e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38867 } } peers { permanent_uuid: "5f143c3e47bf49fba124d1aef220f7b9" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 34301 } attrs { promote: false } } peers { permanent_uuid: "dc4fb41d982b4fbf82dacc12ca19344a" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 44171 } attrs { promote: false } } }
I20260806 09:04:34.716220 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:34.736402 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.725759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46827 (local address 127.0.0.1:45806)
0806 09:04:34.726027 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.726049 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:34.726172 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:34.727213 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:34.727222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:34.727260 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:34.727636 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.727648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.731258 (+ 3610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.731263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.733655 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.733673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.735486 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.735493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.735712 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.735754 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:04:34.735832 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 09:04:34.736005 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":26}
I20260806 09:04:34.737701 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.725882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.27.187.129:46827)
0806 09:04:34.726780 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:34.726788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:34.726810 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:04:34.726895 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:34.726904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:34.727010 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:34.729732 (+ 2722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:34.729973 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.731111 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.733922 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.735253 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.737023 (+ 1770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.737108 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:34.737169 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 09:04:34.737342 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":546,"threads_started":1}
I20260806 09:04:34.739291 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.726641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.27.187.131:45661)
0806 09:04:34.727747 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:34.727757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:34.728514 (+ 757us) server_negotiation.cc:407] Connection header received
0806 09:04:34.728782 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:34.728787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:34.728872 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:34.728950 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:34.729666 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.731004 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.735371 (+ 4367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.738132 (+ 2761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.738252 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.738305 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:34.738812 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 09:04:34.738965 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":779,"threads_started":1}
I20260806 09:04:34.739351 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.726466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45661 (local address 127.0.0.1:37030)
0806 09:04:34.728398 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.728416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:34.728613 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:34.729196 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:34.729202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:34.729230 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:34.729507 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.729513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.731183 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.731188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.735137 (+ 3949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.735155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.738473 (+ 3318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.738478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.738608 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.738631 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:04:34.738776 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 09:04:34.738921 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"mutex_wait_us":3,"thread_start_us":1537,"threads_started":1}
I20260806 09:04:34.740377 3516 tablet_service.cc:1467] Processing CreateTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e (DEFAULT_TABLE table=test-table [id=5a4993e643f44be795bd95cda994adb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:34.740947 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803e71499eea40a59a1aa0b4a9a1786e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:34.742343 3219 tablet_service.cc:1467] Processing CreateTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e (DEFAULT_TABLE table=test-table [id=5a4993e643f44be795bd95cda994adb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:34.742869 3219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803e71499eea40a59a1aa0b4a9a1786e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:34.746235 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.727952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:35413 (local address 127.0.0.1:42616)
0806 09:04:34.736668 (+ 8716us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.736690 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:34.736792 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:34.740811 (+ 4019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:34.740821 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:34.740870 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:34.741149 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.741160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.742803 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.742810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.744109 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.744131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.745374 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.745380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.745577 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.745605 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:04:34.745733 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 09:04:34.745889 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8609,"thread_start_us":13125,"threads_started":1}
I20260806 09:04:34.746266 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.728157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.27.187.130:35413)
0806 09:04:34.740419 (+ 12262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:34.740425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:34.740449 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:04:34.740523 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:34.740528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:34.740625 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:34.740744 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:34.741488 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.742618 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.744317 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.745147 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.745373 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.745456 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:34.745760 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 09:04:34.745910 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12104,"thread_start_us":11964,"threads_started":1}
I20260806 09:04:34.747749 3371 tablet_service.cc:1467] Processing CreateTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e (DEFAULT_TABLE table=test-table [id=5a4993e643f44be795bd95cda994adb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:34.748276 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803e71499eea40a59a1aa0b4a9a1786e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:34.753798 3712 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Bootstrap starting.
I20260806 09:04:34.754885 3710 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Bootstrap starting.
I20260806 09:04:34.756928 3710 tablet_bootstrap.cc:654] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:34.757064 3712 tablet_bootstrap.cc:654] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:34.758693 3711 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Bootstrap starting.
I20260806 09:04:34.761307 3711 tablet_bootstrap.cc:654] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:34.762456 3710 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: No bootstrap required, opened a new log
I20260806 09:04:34.762660 3710 ts_tablet_manager.cc:1397] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 09:04:34.763087 3712 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: No bootstrap required, opened a new log
I20260806 09:04:34.763273 3712 ts_tablet_manager.cc:1397] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 09:04:34.763689 3710 raft_consensus.cc:348] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.763943 3710 raft_consensus.cc:374] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:34.764032 3710 raft_consensus.cc:729] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463fd3157c8e4c57bbbe56e9ec86cb4b, State: Initialized, Role: FOLLOWER
I20260806 09:04:34.764338 3711 tablet_bootstrap.cc:492] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: No bootstrap required, opened a new log
I20260806 09:04:34.764183 3712 raft_consensus.cc:348] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.764429 3712 raft_consensus.cc:374] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:34.764313 3710 consensus_queue.cc:260] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.764483 3711 ts_tablet_manager.cc:1397] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 09:04:34.764523 3712 raft_consensus.cc:729] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a3b54ab59840a6a40662b091b81776, State: Initialized, Role: FOLLOWER
I20260806 09:04:34.764792 3712 consensus_queue.cc:260] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.765328 3711 raft_consensus.cc:348] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.765591 3711 raft_consensus.cc:374] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:34.765703 3711 raft_consensus.cc:729] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3566399ca52e4a689558c05369c24580, State: Initialized, Role: FOLLOWER
I20260806 09:04:34.765986 3711 consensus_queue.cc:260] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.770011 3712 ts_tablet_manager.cc:1428] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 09:04:34.770221 3711 ts_tablet_manager.cc:1428] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 09:04:34.770885 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.770145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:43423)
0806 09:04:34.770501 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.770616 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260806 09:04:34.772817 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.772272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:43139)
0806 09:04:34.772523 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.772619 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 09:04:34.774509 3439 heartbeater.cc:503] Master 127.27.187.189:34301 was elected leader, sending a full tablet report...
I20260806 09:04:34.775571 3710 ts_tablet_manager.cc:1428] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Time spent starting tablet: real 0.013s user 0.000s sys 0.009s
I20260806 09:04:34.777375 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.776849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:46647)
0806 09:04:34.777085 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.777209 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 09:04:34.777779 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.777333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:46273)
0806 09:04:34.777491 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.777607 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 09:04:34.778988 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.778473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:47297)
0806 09:04:34.778719 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.778814 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 09:04:34.780967 3584 heartbeater.cc:503] Master 127.27.187.189:34301 was elected leader, sending a full tablet report...
W20260806 09:04:34.794111 3290 tablet.cc:2369] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:34.962097 3716 raft_consensus.cc:482] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:34.962451 3716 raft_consensus.cc:504] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.964257 3716 leader_election.cc:288] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827), f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:34.972258 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.964735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46827 (local address 127.27.187.131:42385)
0806 09:04:34.964976 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.964999 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:34.965220 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:34.965785 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:34.965792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:34.965827 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:34.966139 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.966153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.967674 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.967680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.969459 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.969475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.971341 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.971349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.971577 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.971614 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:04:34.971746 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 09:04:34.971910 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:04:34.972280 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.964803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:42385 (local address 127.27.187.129:46827)
0806 09:04:34.964994 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:34.965002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:34.965101 (+ 99us) server_negotiation.cc:407] Connection header received
0806 09:04:34.965409 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:34.965417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:34.965526 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:34.965647 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:34.966367 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.967512 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.969659 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.971141 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.971279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.971397 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:34.971763 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 09:04:34.971879 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:04:34.973435 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.965196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:35413 (local address 127.27.187.131:51177)
0806 09:04:34.966004 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.966026 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:34.966166 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:34.966742 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:34.966750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:34.966785 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:34.967079 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.967240 (+ 161us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.968742 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.968747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.970759 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:34.970778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.972706 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.972712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:34.972929 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.972963 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:04:34.973062 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:04:34.973168 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":503,"threads_started":1}
I20260806 09:04:34.973636 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.965430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:51177 (local address 127.27.187.130:35413)
0806 09:04:34.965661 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:34.965668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:34.966311 (+ 643us) server_negotiation.cc:407] Connection header received
0806 09:04:34.966404 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:34.966410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:34.966503 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:34.966622 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:34.967442 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.968562 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.970980 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:34.972291 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:34.972475 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:34.972565 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:34.973150 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 09:04:34.973365 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:04:34.973416 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803e71499eea40a59a1aa0b4a9a1786e" candidate_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3566399ca52e4a689558c05369c24580" is_pre_election: true
I20260806 09:04:34.974169 3239 raft_consensus.cc:2454] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 463fd3157c8e4c57bbbe56e9ec86cb4b in term 0.
I20260806 09:04:34.974234 3391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803e71499eea40a59a1aa0b4a9a1786e" candidate_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a3b54ab59840a6a40662b091b81776" is_pre_election: true
I20260806 09:04:34.974663 3391 raft_consensus.cc:2454] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 463fd3157c8e4c57bbbe56e9ec86cb4b in term 0.
I20260806 09:04:34.974907 3468 leader_election.cc:302] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3566399ca52e4a689558c05369c24580, 463fd3157c8e4c57bbbe56e9ec86cb4b; no voters:
I20260806 09:04:34.975411 3716 raft_consensus.cc:2781] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:34.975574 3716 raft_consensus.cc:482] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:34.975679 3716 raft_consensus.cc:3037] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:34.977869 3716 raft_consensus.cc:504] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.978492 3716 leader_election.cc:288] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [CANDIDATE]: Term 1 election: Requested vote from peers 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827), f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413)
I20260806 09:04:34.979373 3391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803e71499eea40a59a1aa0b4a9a1786e" candidate_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a3b54ab59840a6a40662b091b81776"
I20260806 09:04:34.979351 3239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803e71499eea40a59a1aa0b4a9a1786e" candidate_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3566399ca52e4a689558c05369c24580"
I20260806 09:04:34.979645 3391 raft_consensus.cc:3037] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:34.979674 3239 raft_consensus.cc:3037] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:34.983108 3391 raft_consensus.cc:2454] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463fd3157c8e4c57bbbe56e9ec86cb4b in term 1.
I20260806 09:04:34.983112 3239 raft_consensus.cc:2454] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 463fd3157c8e4c57bbbe56e9ec86cb4b in term 1.
I20260806 09:04:34.983664 3468 leader_election.cc:302] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3566399ca52e4a689558c05369c24580, 463fd3157c8e4c57bbbe56e9ec86cb4b; no voters:
I20260806 09:04:34.984001 3716 raft_consensus.cc:2781] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:34.984601 3716 raft_consensus.cc:686] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 LEADER]: Becoming Leader. State: Replica: 463fd3157c8e4c57bbbe56e9ec86cb4b, State: Running, Role: LEADER
I20260806 09:04:34.985146 3716 consensus_queue.cc:237] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } }
I20260806 09:04:34.985965 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:34.985460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:53009)
0806 09:04:34.985702 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:34.985823 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260806 09:04:34.989322 2909 catalog_manager.cc:5495] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131). New cstate: current_term: 1 leader_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "463fd3157c8e4c57bbbe56e9ec86cb4b" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 } health_report { overall_health: UNKNOWN } } }
I20260806 09:04:35.039894 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.034151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.27.187.131:45661)
0806 09:04:35.034502 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:35.034510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:35.034531 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:04:35.034652 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:35.034657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:35.034780 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:35.034968 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:35.035419 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.036142 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.037471 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.037910 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.038057 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:35.038528 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:35.039178 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:35.039351 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:35.039478 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 09:04:35.039573 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 09:04:35.045087 3391 raft_consensus.cc:1261] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Refusing update from remote peer 463fd3157c8e4c57bbbe56e9ec86cb4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:35.045392 3239 raft_consensus.cc:1261] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Refusing update from remote peer 463fd3157c8e4c57bbbe56e9ec86cb4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:35.046001 3720 consensus_queue.cc:1035] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a3b54ab59840a6a40662b091b81776" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:35.046490 3716 consensus_queue.cc:1035] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3566399ca52e4a689558c05369c24580" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:35.049216 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.048765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:55159)
0806 09:04:35.048969 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.049060 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 09:04:35.049170 3289 heartbeater.cc:503] Master 127.27.187.189:34301 was elected leader, sending a full tablet report...
I20260806 09:04:35.050118 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.049630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:54779)
0806 09:04:35.049808 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.049911 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:04:35.055948 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.055446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.129:57405)
0806 09:04:35.055688 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.055783 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 09:04:35.062294 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.060501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:54301)
0806 09:04:35.061909 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.062045 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1230}
I20260806 09:04:35.062826 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.061294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.130:58265)
0806 09:04:35.062438 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.062540 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1010}
I20260806 09:04:35.067974 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.065647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.27.187.131:51585)
0806 09:04:35.065913 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.066007 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260806 09:04:35.076377 2910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:35.076754 2910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:35.084213 2910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f143c3e47bf49fba124d1aef220f7b9: Sending DeleteTablet for 3 replicas of tablet 803e71499eea40a59a1aa0b4a9a1786e
I20260806 09:04:35.085590 3371 tablet_service.cc:1514] Processing DeleteTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:35 UTC) from {username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:42616
I20260806 09:04:35.085592 3516 tablet_service.cc:1514] Processing DeleteTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:35 UTC) from {username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:37030
I20260806 09:04:35.085657 3219 tablet_service.cc:1514] Processing DeleteTablet for tablet 803e71499eea40a59a1aa0b4a9a1786e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:35 UTC) from {username='oryx', principal='oryx/127.27.187.189@KRBTEST.COM'} at 127.0.0.1:45806
I20260806 09:04:35.086508 3732 tablet_replica.cc:330] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: stopping tablet replica
I20260806 09:04:35.086823 3732 raft_consensus.cc:2229] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:35.087833 3732 raft_consensus.cc:2258] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:35.088240 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 3082
I20260806 09:04:35.088847 3733 tablet_replica.cc:330] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: stopping tablet replica
I20260806 09:04:35.089075 3734 tablet_replica.cc:330] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: stopping tablet replica
I20260806 09:04:35.089076 3733 raft_consensus.cc:2229] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:35.089358 3734 raft_consensus.cc:2229] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:35.089776 3734 raft_consensus.cc:2258] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:35.089819 3733 raft_consensus.cc:2258] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:35.090997 3733 ts_tablet_manager.cc:1905] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:35.091822 3734 ts_tablet_manager.cc:1905] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:35.099632 3733 ts_tablet_manager.cc:1918] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:35.099717 3732 ts_tablet_manager.cc:1905] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:35.099813 3733 log.cc:1196] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/803e71499eea40a59a1aa0b4a9a1786e
I20260806 09:04:35.100379 3733 ts_tablet_manager.cc:1939] T 803e71499eea40a59a1aa0b4a9a1786e P 463fd3157c8e4c57bbbe56e9ec86cb4b: Deleting consensus metadata
I20260806 09:04:35.101795 2897 catalog_manager.cc:4841] TS 463fd3157c8e4c57bbbe56e9ec86cb4b (127.27.187.131:45661): tablet 803e71499eea40a59a1aa0b4a9a1786e (table test-table [id=5a4993e643f44be795bd95cda994adb1]) successfully deleted
I20260806 09:04:35.102864 3734 ts_tablet_manager.cc:1918] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:35.103049 3734 log.cc:1196] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/803e71499eea40a59a1aa0b4a9a1786e
I20260806 09:04:35.103647 3734 ts_tablet_manager.cc:1939] T 803e71499eea40a59a1aa0b4a9a1786e P 3566399ca52e4a689558c05369c24580: Deleting consensus metadata
I20260806 09:04:35.104943 2896 catalog_manager.cc:4841] TS 3566399ca52e4a689558c05369c24580 (127.27.187.129:46827): tablet 803e71499eea40a59a1aa0b4a9a1786e (table test-table [id=5a4993e643f44be795bd95cda994adb1]) successfully deleted
I20260806 09:04:35.106021 3732 ts_tablet_manager.cc:1918] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:35.106185 3732 log.cc:1196] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/803e71499eea40a59a1aa0b4a9a1786e
I20260806 09:04:35.106653 3732 ts_tablet_manager.cc:1939] T 803e71499eea40a59a1aa0b4a9a1786e P f3a3b54ab59840a6a40662b091b81776: Deleting consensus metadata
I20260806 09:04:35.107790 2896 catalog_manager.cc:4841] TS f3a3b54ab59840a6a40662b091b81776 (127.27.187.130:35413): tablet 803e71499eea40a59a1aa0b4a9a1786e (table test-table [id=5a4993e643f44be795bd95cda994adb1]) successfully deleted
I20260806 09:04:35.119246 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.089758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.27.187.129:46827)
0806 09:04:35.089971 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:35.089979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:35.100946 (+ 10967us) server_negotiation.cc:407] Connection header received
0806 09:04:35.101504 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:35.101510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:35.101604 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:35.101730 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:35.104038 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.105234 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.106845 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.107334 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.107504 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:35.111381 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:35.111402 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:35.111419 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:35.111474 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:35.115623 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.116616 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.116626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.116629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.116811 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.117308 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.117320 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.117323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.117771 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:35.118122 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:35.118495 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 09:04:35.118836 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:04:35.120419 3282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:35.257781 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.250502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37670)
0806 09:04:35.257328 (+ 6826us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.257465 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6670,"thread_start_us":6543,"threads_started":1}
W20260806 09:04:35.258087 2895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260806 09:04:35.259670 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.259223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38867 (local address 127.0.0.1:37678)
0806 09:04:35.259410 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:35.259502 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:38867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 09:04:35.272975 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 3082
I20260806 09:04:35.300997 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 3300
I20260806 09:04:35.317690 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.301987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.27.187.130:35413)
0806 09:04:35.302174 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:35.302183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:35.302206 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:04:35.302440 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:35.302446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:35.302537 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:35.302650 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:35.304128 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.304888 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.306173 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.306604 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.306814 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:35.310552 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:35.310570 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:35.310586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:35.310630 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:35.314126 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.315162 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.315172 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.315175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.315341 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.315820 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.315827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.315835 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.316279 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:35.316658 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:35.317006 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 09:04:35.317290 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:04:35.318478 3432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:35.421233 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 3300
I20260806 09:04:35.449316 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 3451
I20260806 09:04:35.468391 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:35.450330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.27.187.131:45661)
0806 09:04:35.450528 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:35.450533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:35.450554 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:04:35.450796 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:35.450800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:35.450908 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:35.451005 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:35.453009 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.453969 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.455983 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:35.456561 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:35.456688 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:35.460462 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:35.460483 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:35.460500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:35.460551 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:35.464390 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.465485 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.465494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.465497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.465722 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:35.466254 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:35.466260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:35.466263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:35.466886 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:35.467212 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:35.467594 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 09:04:35.467794 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:04:35.469172 3577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:35.613480 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 3451
I20260806 09:04:35.638422 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2766
I20260806 09:04:35.639842 2942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:35.775219 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2766
I20260806 09:04:35.800774 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 2866
I20260806 09:04:35.802002 3135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:35.924789 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 2866
2026-08-06T09:04:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (15373 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:36 dist-test-slave-8tdl krb5kdc[3745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:36 dist-test-slave-8tdl krb5kdc[3745](info): set up 2 sockets
Aug 06 09:04:36 dist-test-slave-8tdl krb5kdc[3745](info): commencing operation
krb5kdc: starting...
W20260806 09:04:37.118850 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:37 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab.
Entry for principal oryx/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab.
I20260806 09:04:37.417052 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43253
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab
--principal=oryx/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:37.792654 3758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:37.793032 3758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:37.793275 3758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:37.802585 3758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:37.802873 3758 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:04:37.802954 3758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:37.803058 3758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:37.803138 3758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:37.803323 3758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:37.814229 3758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:43253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.190.keytab
--principal=oryx/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:37.815722 3758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:37.817157 3758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:37.832249 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:37.832333 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:37.832365 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:37.832837 3758 server_base.cc:1034] running on GCE node
I20260806 09:04:37.833978 3758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:37.835862 3758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:37.837214 3758 hybrid_clock.cc:648] HybridClock initialized: now 1786007077837038 us; error 64 us; skew 500 ppm
I20260806 09:04:37.837670 3758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:37 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:37.841538 3758 init.cc:377] Logged in from keytab as oryx/127.27.187.190@KRBTEST.COM (short username oryx)
I20260806 09:04:37.843691 3758 webserver.cc:460] Webserver started at http://127.27.187.190:45975/ using document root <none> and password file <none>
I20260806 09:04:37.844578 3758 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:37.844686 3758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:37.845070 3758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:37.848783 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1"
format_stamp: "Formatted at 2026-08-06 09:04:37 on dist-test-slave-8tdl"
server_key: "53bfdc8f72e81b8dc54d85d0039b275b"
server_key_iv: "63471cf952e6ecd64f56bc010d3e6728"
server_key_version: "encryptionkey@0"
I20260806 09:04:37.849781 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1"
format_stamp: "Formatted at 2026-08-06 09:04:37 on dist-test-slave-8tdl"
server_key: "53bfdc8f72e81b8dc54d85d0039b275b"
server_key_iv: "63471cf952e6ecd64f56bc010d3e6728"
server_key_version: "encryptionkey@0"
I20260806 09:04:37.856371 3758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:04:37.860944 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:37.862818 3758 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:04:37.863058 3758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1"
format_stamp: "Formatted at 2026-08-06 09:04:37 on dist-test-slave-8tdl"
server_key: "53bfdc8f72e81b8dc54d85d0039b275b"
server_key_iv: "63471cf952e6ecd64f56bc010d3e6728"
server_key_version: "encryptionkey@0"
I20260806 09:04:37.863333 3758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:37.893522 3758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:37.898739 3758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:37.899080 3758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:37.924672 3758 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:43253
I20260806 09:04:37.924690 3835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:43253 every 8 connection(s)
I20260806 09:04:37.926213 3758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:37.928098 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 3758
I20260806 09:04:37.928355 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:37.929039 28398 external_mini_cluster.cc:1428] Setting key 7995f6a558c231a7ef67affa29b10d71
I20260806 09:04:37.931692 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:37.946489 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Bootstrap starting.
Aug 06 09:04:37 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:37.950984 3836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:37.952574 3836 log.cc:824] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:37.956727 3836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: No bootstrap required, opened a new log
I20260806 09:04:37.958214 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:37.931377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.27.187.190:43253)
0806 09:04:37.932288 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:37.932305 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:37.932360 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:04:37.934862 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:37.934895 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:37.935599 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:37.936139 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:37.937879 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:37.940182 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:37.941654 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:37.942846 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:37.943048 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:37.948020 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:37.948039 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:37.948064 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:37.948136 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:37.952582 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:37.953762 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:37.953770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:37.953777 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:37.953972 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:37.954495 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:37.954501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:37.954504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:37.955317 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:37.955630 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:37.956132 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 09:04:37.956666 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":399,"threads_started":1}
I20260806 09:04:37.965996 3836 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } }
I20260806 09:04:37.966475 3836 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:37.966619 3836 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb71f3ba0db40f997bd30d2b3dc9ff1, State: Initialized, Role: FOLLOWER
I20260806 09:04:37.967432 3836 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } }
I20260806 09:04:37.967796 3836 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:37.967945 3836 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:37.968147 3836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:37.970583 3836 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } }
I20260806 09:04:37.971184 3836 leader_election.cc:302] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb71f3ba0db40f997bd30d2b3dc9ff1; no voters:
I20260806 09:04:37.971871 3836 leader_election.cc:288] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:37.972059 3841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:37.973701 3841 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [term 1 LEADER]: Becoming Leader. State: Replica: feb71f3ba0db40f997bd30d2b3dc9ff1, State: Running, Role: LEADER
I20260806 09:04:37.974300 3836 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:37.974334 3841 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } }
I20260806 09:04:37.977802 3842 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } } }
I20260806 09:04:37.977847 3844 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb71f3ba0db40f997bd30d2b3dc9ff1. Latest consensus state: current_term: 1 leader_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb71f3ba0db40f997bd30d2b3dc9ff1" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43253 } } }
I20260806 09:04:37.978366 3844 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:37.978366 3842 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:37.979923 3850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:37.986310 3850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:37.996340 3850 catalog_manager.cc:1347] Generated new cluster ID: 948e9bb4d09f42268b69939c750b9725
I20260806 09:04:37.996497 3850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:38.017449 3850 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:38.019403 3850 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:38.029259 3850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Generated new TSK 0
I20260806 09:04:38.030295 3850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
I20260806 09:04:38.099826 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43253
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:38.462982 3864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:38.463302 3864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:38.463405 3864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:38.471341 3864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:38.471470 3864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:38.471557 3864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:04:38.471640 3864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:38.482426 3864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:43253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:38.483883 3864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:38.485347 3864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:38.500001 3869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:38.500023 3870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:38.499985 3872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:38.506809 3864 server_base.cc:1034] running on GCE node
I20260806 09:04:38.507761 3864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:38.509097 3864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:38.510375 3864 hybrid_clock.cc:648] HybridClock initialized: now 1786007078510298 us; error 43 us; skew 500 ppm
I20260806 09:04:38.510875 3864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:38 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007078, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:38.515129 3864 init.cc:377] Logged in from keytab as oryx/127.27.187.129@KRBTEST.COM (short username oryx)
I20260806 09:04:38.517035 3864 webserver.cc:460] Webserver started at http://127.27.187.129:33519/ using document root <none> and password file <none>
I20260806 09:04:38.517840 3864 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:38.517937 3864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:38.518327 3864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:38.521569 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "6f553021ab414b35b8c0a38009352052"
format_stamp: "Formatted at 2026-08-06 09:04:38 on dist-test-slave-8tdl"
server_key: "b33045cd78d82526009674e162b95fb1"
server_key_iv: "1d121aa81cc5961645b83629648d310b"
server_key_version: "encryptionkey@0"
I20260806 09:04:38.522477 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "6f553021ab414b35b8c0a38009352052"
format_stamp: "Formatted at 2026-08-06 09:04:38 on dist-test-slave-8tdl"
server_key: "b33045cd78d82526009674e162b95fb1"
server_key_iv: "1d121aa81cc5961645b83629648d310b"
server_key_version: "encryptionkey@0"
I20260806 09:04:38.528366 3864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:04:38.532657 3879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:38.534371 3864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 09:04:38.534595 3864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "6f553021ab414b35b8c0a38009352052"
format_stamp: "Formatted at 2026-08-06 09:04:38 on dist-test-slave-8tdl"
server_key: "b33045cd78d82526009674e162b95fb1"
server_key_iv: "1d121aa81cc5961645b83629648d310b"
server_key_version: "encryptionkey@0"
I20260806 09:04:38.534911 3864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:38.572762 3864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:38.578120 3864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:38.578486 3864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:38.581877 3864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:38.581974 3864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:38.582115 3864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:38.582203 3864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:04:38 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007078, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:38.609476 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:38.582912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34119 (local address 127.27.187.190:43253)
0806 09:04:38.583406 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:38.583411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:38.584719 (+ 1308us) server_negotiation.cc:407] Connection header received
0806 09:04:38.587267 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:38.587275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:38.587396 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:38.587487 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:38.590314 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.591314 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.594095 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.594950 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.595120 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:38.600559 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:38.600571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:38.600580 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:38.600644 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:38.604852 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:38.606108 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:38.606117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:38.606120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:38.606279 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:38.606943 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:38.606950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:38.606957 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:38.607426 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:38.607606 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:38.608531 (+ 925us) server_negotiation.cc:299] Negotiation successful
0806 09:04:38.608881 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
I20260806 09:04:38.610044 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:38.583641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.129:34119)
0806 09:04:38.584540 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:38.584592 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:38.586884 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:38.588100 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:38.588109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:38.588966 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:38.590105 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:38.590128 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.591486 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.591491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:38.593857 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:38.593873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.595261 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.595272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:38.595634 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:38.596613 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:38.596642 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:38.600227 (+ 3585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:38.605020 (+ 4793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:38.605034 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:38.605067 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:38.605882 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:38.606430 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:38.606436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:38.606440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:38.606781 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:38.607632 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:38.607644 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:38.608065 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 09:04:38.608316 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 09:04:38.608772 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":415,"threads_started":1}
I20260806 09:04:38.633790 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:38.616906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:51381 (local address 127.27.187.190:43253)
0806 09:04:38.617175 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:38.617185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:38.617299 (+ 114us) server_negotiation.cc:407] Connection header received
0806 09:04:38.617594 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:38.617599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:38.617679 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:38.617788 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:38.619466 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.620853 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.623056 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.623526 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.623704 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:38.626119 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:38.626125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:38.626132 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:38.626176 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:38.629905 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:38.631305 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:38.631310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:38.631312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:38.631458 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:38.632027 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:38.632030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:38.632032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:38.632421 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:38.632603 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:38.633259 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 09:04:38.633471 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:04:38.634310 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:38.616498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.129:51381)
0806 09:04:38.617145 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:38.617166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:38.617415 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:38.617837 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:38.617842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:38.618666 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:38.619224 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:38.619246 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.621055 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.621064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:38.622872 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:38.622891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:38.623771 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:38.623779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:38.623990 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:38.624831 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:38.624850 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:38.625887 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:38.630147 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:38.630156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:38.630165 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:38.631123 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:38.631597 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:38.631602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:38.631609 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:38.631846 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:38.632582 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:38.632595 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:38.632883 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 09:04:38.633092 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 09:04:38.633619 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":478,"threads_started":1}
I20260806 09:04:38.641197 3864 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:33245
I20260806 09:04:38.641295 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:33245 every 8 connection(s)
I20260806 09:04:38.642635 3864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:38.645699 4004 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43253
I20260806 09:04:38.645969 4004 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:38.646664 4004 heartbeater.cc:511] Master 127.27.187.190:43253 requested a full tablet report, sending...
I20260806 09:04:38.649358 3791 ts_manager.cc:194] Registered new tserver with Master: 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245)
I20260806 09:04:38.651064 3791 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:34119
I20260806 09:04:38.651391 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 3864
I20260806 09:04:38.651674 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:38.652542 28398 external_mini_cluster.cc:1428] Setting key 991a6fe752f20f0c2abc5ecb4893759b
WARNING: no policy specified for oryx/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
I20260806 09:04:38.720224 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43253
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:39.093919 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:39.094307 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:39.094489 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:39.102530 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:39.102741 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:39.102898 4008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:04:39.103049 4008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:39.114269 4008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:43253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:39.115787 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:39.117272 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:39.132941 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:39.132953 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:39.132984 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:39.133774 4008 server_base.cc:1034] running on GCE node
I20260806 09:04:39.134750 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:39.135844 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:39.137077 4008 hybrid_clock.cc:648] HybridClock initialized: now 1786007079136991 us; error 63 us; skew 500 ppm
I20260806 09:04:39.137445 4008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:39.146646 4008 init.cc:377] Logged in from keytab as oryx/127.27.187.130@KRBTEST.COM (short username oryx)
I20260806 09:04:39.148583 4008 webserver.cc:460] Webserver started at http://127.27.187.130:35083/ using document root <none> and password file <none>
I20260806 09:04:39.149488 4008 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:39.149662 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:39.150019 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:39.153262 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "2e84e1121920432ab107c553d4e669a8"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "326459de2b53c2eba19076787202f92e"
server_key_iv: "0122f355585cdf348f4aedb53a935211"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.154137 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "2e84e1121920432ab107c553d4e669a8"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "326459de2b53c2eba19076787202f92e"
server_key_iv: "0122f355585cdf348f4aedb53a935211"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.160303 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:04:39.164322 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:39.165839 4008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:04:39.166033 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "2e84e1121920432ab107c553d4e669a8"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "326459de2b53c2eba19076787202f92e"
server_key_iv: "0122f355585cdf348f4aedb53a935211"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.166333 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:39.186400 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:39.191532 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:39.191838 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:39.194926 4008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:39.195046 4008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:39.195184 4008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:39.195308 4008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:39.222543 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.195483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:44691 (local address 127.27.187.190:43253)
0806 09:04:39.196330 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.196335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.197352 (+ 1017us) server_negotiation.cc:407] Connection header received
0806 09:04:39.199548 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.199554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.199665 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.199788 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.202171 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.203358 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.206952 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.207767 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.207933 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.213672 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:39.213681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:39.213685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:39.213755 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:39.218173 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.219637 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.219645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.219651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.219828 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.220386 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.220392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.220396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.220743 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:39.221005 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:39.221805 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 09:04:39.222109 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":651,"threads_started":1}
I20260806 09:04:39.223730 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.196229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.130:44691)
0806 09:04:39.197168 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:39.197219 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:39.199235 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:39.200051 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:39.200059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:39.200923 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:39.201960 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.201993 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.203616 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.203629 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.206698 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.206738 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.208083 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.208093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.208485 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.209391 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:39.209424 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:39.213211 (+ 3787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:39.218414 (+ 5203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.218425 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.218452 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.219436 (+ 984us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.219982 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.219987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.219990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.220242 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.220893 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:39.220903 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:39.221372 (+ 469us) client_negotiation.cc:769] Sending connection context
0806 09:04:39.221676 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 09:04:39.222314 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260806 09:04:39.250626 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.230645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:35717 (local address 127.27.187.190:43253)
0806 09:04:39.230952 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.230963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.231925 (+ 962us) server_negotiation.cc:407] Connection header received
0806 09:04:39.232185 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.232191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.232325 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.232453 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.234257 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.235447 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.238374 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.238932 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.239080 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.242102 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:39.242112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:39.242120 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:39.242173 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:39.246636 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.248054 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.248059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.248062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.248201 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.248768 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.248776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.248781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.249370 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:39.249531 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:39.250015 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 09:04:39.250275 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 09:04:39.250782 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.230649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.130:35717)
0806 09:04:39.231801 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:39.231820 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:39.232012 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:39.232515 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:39.232520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:39.233339 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:39.234030 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.234051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.235648 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.235657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.238171 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.238197 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.239176 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.239183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.239468 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.240426 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:39.240456 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:39.241774 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:39.246923 (+ 5149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.246931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.246939 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.247840 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.248420 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.248428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.248434 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.248610 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.249520 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:39.249525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:39.249735 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 09:04:39.249906 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 09:04:39.250277 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":842,"threads_started":1}
I20260806 09:04:39.251396 4008 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36987
I20260806 09:04:39.251484 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36987 every 8 connection(s)
I20260806 09:04:39.253141 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:39.255551 4145 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43253
I20260806 09:04:39.255946 4145 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:39.256641 4145 heartbeater.cc:511] Master 127.27.187.190:43253 requested a full tablet report, sending...
I20260806 09:04:39.258102 3791 ts_manager.cc:194] Registered new tserver with Master: 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:39.259064 3791 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:44691
I20260806 09:04:39.262027 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4008
I20260806 09:04:39.262293 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:39.262811 28398 external_mini_cluster.cc:1428] Setting key 184e73f40179e8c18bba5c525828d304
WARNING: no policy specified for oryx/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
I20260806 09:04:39.327278 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43253
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:39.654850 4004 heartbeater.cc:503] Master 127.27.187.190:43253 was elected leader, sending a full tablet report...
W20260806 09:04:39.689502 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:39.689826 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:39.689926 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:39.697763 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:39.697882 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:39.697970 4149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:04:39.698053 4149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:39.708643 4149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.187.190:43253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:39.710031 4149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:39.711541 4149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:39.730916 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:39.730988 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:39.732815 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:39.733547 4149 server_base.cc:1034] running on GCE node
I20260806 09:04:39.734386 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:39.735586 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:39.737125 4149 hybrid_clock.cc:648] HybridClock initialized: now 1786007079736950 us; error 69 us; skew 500 ppm
I20260806 09:04:39.737757 4149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:39.741815 4149 init.cc:377] Logged in from keytab as oryx/127.27.187.131@KRBTEST.COM (short username oryx)
I20260806 09:04:39.743609 4149 webserver.cc:460] Webserver started at http://127.27.187.131:46283/ using document root <none> and password file <none>
I20260806 09:04:39.744356 4149 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:39.744455 4149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:39.744755 4149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:39.748127 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "c248f66b886b4432b0d59b80d53643d1"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "eba0fc8f9ffdcff375fe63049dc81e85"
server_key_iv: "b79d90e0610a1330d222b93e58cbfbb1"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.749037 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "c248f66b886b4432b0d59b80d53643d1"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "eba0fc8f9ffdcff375fe63049dc81e85"
server_key_iv: "b79d90e0610a1330d222b93e58cbfbb1"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.755249 4149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 09:04:39.759752 4164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:39.761482 4149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:04:39.761682 4149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "c248f66b886b4432b0d59b80d53643d1"
format_stamp: "Formatted at 2026-08-06 09:04:39 on dist-test-slave-8tdl"
server_key: "eba0fc8f9ffdcff375fe63049dc81e85"
server_key_iv: "b79d90e0610a1330d222b93e58cbfbb1"
server_key_version: "encryptionkey@0"
I20260806 09:04:39.761922 4149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:39.820924 4149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:39.826215 4149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:39.826491 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:39.829937 4149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:39.830181 4149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:04:39.830411 4149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:39.830504 4149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:39.861904 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.831093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:54201 (local address 127.27.187.190:43253)
0806 09:04:39.831725 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.831735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.833081 (+ 1346us) server_negotiation.cc:407] Connection header received
0806 09:04:39.835481 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.835486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.835586 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.835727 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.838513 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.839751 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.843970 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.844772 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.844958 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.852583 (+ 7625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:39.852594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:39.852602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:39.852659 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:39.856268 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.857689 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.857699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.857706 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.857914 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.858772 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.858777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.858780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.859283 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:39.859478 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:39.860995 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0806 09:04:39.861361 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":356,"threads_started":1}
I20260806 09:04:39.864419 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.831875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.131:54201)
0806 09:04:39.832848 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:39.832897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:39.835136 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:39.836051 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:39.836067 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:39.836999 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:39.838067 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.838116 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.840013 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.840020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.843690 (+ 3670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.843722 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.845175 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.845188 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.845732 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.846921 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:39.846980 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:39.852092 (+ 5112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:39.856490 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.856508 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.856536 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.857479 (+ 943us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.858102 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.858115 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.858121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.858548 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.859461 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:39.859478 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:39.860353 (+ 875us) client_negotiation.cc:769] Sending connection context
0806 09:04:39.860823 (+ 470us) client_negotiation.cc:240] Negotiation successful
0806 09:04:39.861748 (+ 925us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":439,"threads_started":1}
I20260806 09:04:39.877527 4149 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:41299
I20260806 09:04:39.877645 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:41299 every 8 connection(s)
I20260806 09:04:39.879369 4149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:39.881891 4287 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43253
I20260806 09:04:39.882140 4287 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:39.882893 4287 heartbeater.cc:511] Master 127.27.187.190:43253 requested a full tablet report, sending...
I20260806 09:04:39.884548 3791 ts_manager.cc:194] Registered new tserver with Master: c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299)
I20260806 09:04:39.885813 3791 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:54201
I20260806 09:04:39.889274 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4149
I20260806 09:04:39.889468 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:39.889928 28398 external_mini_cluster.cc:1428] Setting key c18ad6a5b5d7e5d95fd4492eb7e234af
I20260806 09:04:39.890578 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.873022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:51173 (local address 127.27.187.190:43253)
0806 09:04:39.873210 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.873215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.874094 (+ 879us) server_negotiation.cc:407] Connection header received
0806 09:04:39.874405 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.874413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.874515 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.874623 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.876363 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.877628 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.879641 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.880395 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.880551 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.884213 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:39.884218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:39.884222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:39.884266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:39.887718 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.888479 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.888485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.888489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.888641 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.889059 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.889064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.889069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.889370 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:39.889538 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:39.889942 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 09:04:39.890184 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:04:39.890656 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.873066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.27.187.131:51173)
0806 09:04:39.873956 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:39.873985 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:39.874217 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:39.874926 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:39.874936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:39.875677 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:39.876126 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.876145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.877828 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.877835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.879463 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:39.879478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.880701 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.880713 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:39.881008 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.882910 (+ 1902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:39.882930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:39.883939 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:39.887877 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.887882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.887887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.888341 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.888764 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:39.888769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:39.888773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:39.888931 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:39.889497 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:39.889502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:39.889686 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 09:04:39.889835 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 09:04:39.890174 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":457,"threads_started":1}
I20260806 09:04:39.893682 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:04:39.969908 4173 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:39.956588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.27.187.190:43253)
0806 09:04:39.956853 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.956862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.957047 (+ 185us) server_negotiation.cc:407] Connection header received
0806 09:04:39.957348 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.957352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.957426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.957525 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.959525 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.960415 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.961906 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.962364 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.962480 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.969455 (+ 6975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38112: BlockingRecv error: recv got EOF from 127.0.0.1:38112 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 09:04:39 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:39.992414 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:39.974418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.27.187.190:43253)
0806 09:04:39.974609 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:39.974616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:39.974934 (+ 318us) server_negotiation.cc:407] Connection header received
0806 09:04:39.975133 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:39.975137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:39.975223 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:39.975371 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:39.977148 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.978050 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.979834 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:39.980504 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:39.980627 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:39.985181 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:39.985188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:39.985192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:39.985240 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:39.988703 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.989814 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.989819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.989822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.989988 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:39.990632 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:39.990638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:39.990642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:39.991164 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:39.991449 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:39.991830 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 09:04:39.992082 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:04:39.998317 3790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:40.007000 3790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:40.041713 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.025523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41299 (local address 127.0.0.1:41228)
0806 09:04:40.026855 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.026905 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.027113 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.029518 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.029531 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.029577 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.030150 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.030169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.033195 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.033203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.036805 (+ 3602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.036834 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.040250 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.040261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.040663 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.040745 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.040929 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.041249 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":761,"threads_started":1}
I20260806 09:04:40.042351 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.025881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.27.187.131:41299)
0806 09:04:40.028164 (+ 2283us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.028174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.028219 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:04:40.028340 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.028352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.028840 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.029090 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.030489 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.032942 (+ 2453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.037119 (+ 4177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.039437 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.039753 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.040031 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.041514 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.041723 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":1809,"threads_started":1}
I20260806 09:04:40.048198 4216 tablet_service.cc:1467] Processing CreateTablet for tablet 2844eec50383493da8428ea3c7452e9d (DEFAULT_TABLE table=test-table [id=54ac73bfcc284b67a3a84cb796557571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.050275 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.029203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.27.187.130:36987)
0806 09:04:40.039615 (+ 10412us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.039626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.041811 (+ 2185us) server_negotiation.cc:407] Connection header received
0806 09:04:40.041939 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.041947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.042263 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.042500 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.043225 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.044293 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.047688 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.049114 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.049350 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.049571 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.049806 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.049979 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9871,"thread_start_us":10158,"threads_started":1}
I20260806 09:04:40.050325 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.028675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36987 (local address 127.0.0.1:43976)
0806 09:04:40.041522 (+ 12847us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.041543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.041668 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.042635 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.042640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.042719 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.043027 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.043035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.044514 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.044518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.047421 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.047444 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.049379 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.049386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.049563 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.049607 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.049726 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.049894 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12697,"mutex_wait_us":61,"thread_start_us":12533,"threads_started":1}
I20260806 09:04:40.050393 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2844eec50383493da8428ea3c7452e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.052703 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.025523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33245 (local address 127.0.0.1:58466)
0806 09:04:40.028670 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.028703 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.028881 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.040895 (+ 12014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.040901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.040931 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.041450 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.041464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.044447 (+ 2983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.044455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.048165 (+ 3710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.048182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.050835 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.050843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.051033 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.051558 (+ 525us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.051776 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.052154 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2711,"mutex_wait_us":1027,"thread_start_us":2206,"threads_started":1}
I20260806 09:04:40.053776 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.025929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.27.187.129:33245)
0806 09:04:40.040070 (+ 14141us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.040079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.040115 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:04:40.040244 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.040253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.040629 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.041709 (+ 1080us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.042182 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.044289 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.048519 (+ 4230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.050197 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.050849 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.051043 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.053184 (+ 2141us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.053439 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13972,"thread_start_us":14881,"threads_started":1}
I20260806 09:04:40.055716 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 2844eec50383493da8428ea3c7452e9d (DEFAULT_TABLE table=test-table [id=54ac73bfcc284b67a3a84cb796557571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.057137 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2844eec50383493da8428ea3c7452e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.057984 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 2844eec50383493da8428ea3c7452e9d (DEFAULT_TABLE table=test-table [id=54ac73bfcc284b67a3a84cb796557571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.059672 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2844eec50383493da8428ea3c7452e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.068630 4312 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Bootstrap starting.
I20260806 09:04:40.072651 4313 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Bootstrap starting.
I20260806 09:04:40.074643 4312 tablet_bootstrap.cc:654] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.076092 4312 log.cc:824] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:40.077122 4313 tablet_bootstrap.cc:654] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.078779 4315 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Bootstrap starting.
I20260806 09:04:40.079028 4313 log.cc:824] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:40.081403 4312 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: No bootstrap required, opened a new log
I20260806 09:04:40.081823 4312 ts_tablet_manager.cc:1397] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260806 09:04:40.083434 4315 tablet_bootstrap.cc:654] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.084916 4315 log.cc:824] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:40.088425 4315 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: No bootstrap required, opened a new log
I20260806 09:04:40.088718 4313 tablet_bootstrap.cc:492] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: No bootstrap required, opened a new log
I20260806 09:04:40.088845 4315 ts_tablet_manager.cc:1397] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 09:04:40.089048 4313 ts_tablet_manager.cc:1397] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260806 09:04:40.093827 4312 raft_consensus.cc:348] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.094360 4312 raft_consensus.cc:374] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.094468 4312 raft_consensus.cc:729] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.095257 4312 consensus_queue.cc:260] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.097565 4287 heartbeater.cc:503] Master 127.27.187.190:43253 was elected leader, sending a full tablet report...
I20260806 09:04:40.098214 4312 ts_tablet_manager.cc:1428] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 09:04:40.098459 4315 raft_consensus.cc:348] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.099017 4315 raft_consensus.cc:374] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.099139 4315 raft_consensus.cc:729] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.099921 4315 consensus_queue.cc:260] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.100274 4313 raft_consensus.cc:348] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.100773 4313 raft_consensus.cc:374] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.100894 4313 raft_consensus.cc:729] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e84e1121920432ab107c553d4e669a8, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.101518 4313 consensus_queue.cc:260] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.102229 4315 ts_tablet_manager.cc:1428] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260806 09:04:40.104149 4313 ts_tablet_manager.cc:1428] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260806 09:04:40.104589 4145 heartbeater.cc:503] Master 127.27.187.190:43253 was elected leader, sending a full tablet report...
W20260806 09:04:40.132562 4289 tablet.cc:2369] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:04:40.145993 4005 tablet.cc:2369] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:04:40.256614 4146 tablet.cc:2369] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:04:40.289945 4319 raft_consensus.cc:482] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:40.290328 4319 raft_consensus.cc:504] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.292373 4319 leader_election.cc:288] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.300544 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.292957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:42899 (local address 127.27.187.130:36987)
0806 09:04:40.293151 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.293157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.294051 (+ 894us) server_negotiation.cc:407] Connection header received
0806 09:04:40.294254 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.294258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.294321 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.294436 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.295288 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.296127 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.298263 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.299550 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.299743 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.299823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.300118 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.300264 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:04:40.300655 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.292950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36987 (local address 127.27.187.129:42899)
0806 09:04:40.293887 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.293903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.294082 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.294480 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.294487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.294514 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.295024 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.295035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.296302 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.296306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.297923 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.297979 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.299798 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.299803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.299939 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.299967 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.300055 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.300157 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":24,"thread_start_us":602,"threads_started":1}
I20260806 09:04:40.300817 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.292885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:54169 (local address 127.27.187.131:41299)
0806 09:04:40.293118 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.293125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.293982 (+ 857us) server_negotiation.cc:407] Connection header received
0806 09:04:40.294208 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.294214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.294290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.294365 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.295149 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.295892 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.298310 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.299569 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.299743 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.299823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.300378 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.300528 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:04:40.300987 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.292890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41299 (local address 127.27.187.129:54169)
0806 09:04:40.293842 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.293868 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.294045 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.294422 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.294441 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.294477 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.294995 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.295010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.296055 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.296061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.298057 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.298073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.299994 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.299999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.300187 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.300221 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.300323 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.300465 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":168,"thread_start_us":529,"threads_started":1}
I20260806 09:04:40.301985 4237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2844eec50383493da8428ea3c7452e9d" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c248f66b886b4432b0d59b80d53643d1" is_pre_election: true
I20260806 09:04:40.302150 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2844eec50383493da8428ea3c7452e9d" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8" is_pre_election: true
I20260806 09:04:40.302537 4237 raft_consensus.cc:2454] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 0.
I20260806 09:04:40.302779 4100 raft_consensus.cc:2454] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 0.
I20260806 09:04:40.303344 3882 leader_election.cc:302] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f553021ab414b35b8c0a38009352052, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.303791 4319 raft_consensus.cc:2781] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:40.303951 4319 raft_consensus.cc:482] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:40.304061 4319 raft_consensus.cc:3037] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.306371 4319 raft_consensus.cc:504] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.307147 4319 leader_election.cc:288] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 election: Requested vote from peers c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.307816 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2844eec50383493da8428ea3c7452e9d" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8"
I20260806 09:04:40.307816 4237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2844eec50383493da8428ea3c7452e9d" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c248f66b886b4432b0d59b80d53643d1"
I20260806 09:04:40.308068 4237 raft_consensus.cc:3037] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.308068 4100 raft_consensus.cc:3037] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.311632 4100 raft_consensus.cc:2454] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 1.
I20260806 09:04:40.311633 4237 raft_consensus.cc:2454] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 1.
I20260806 09:04:40.312259 3882 leader_election.cc:302] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f553021ab414b35b8c0a38009352052, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.312641 4319 raft_consensus.cc:2781] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:40.313241 4319 raft_consensus.cc:686] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 1 LEADER]: Becoming Leader. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Running, Role: LEADER
I20260806 09:04:40.313803 4319 consensus_queue.cc:237] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } }
I20260806 09:04:40.319218 3791 catalog_manager.cc:5495] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f553021ab414b35b8c0a38009352052 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "6f553021ab414b35b8c0a38009352052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } health_report { overall_health: UNKNOWN } } }
I20260806 09:04:40.343924 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.337740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.27.187.129:33245)
0806 09:04:40.337941 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.337946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.338034 (+ 88us) server_negotiation.cc:407] Connection header received
0806 09:04:40.338351 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.338356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.338457 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.338596 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:40.339126 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.339921 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.341528 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.341941 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.342039 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.342524 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:40.343254 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:40.343373 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.343519 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.343592 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:04:40.353572 4100 raft_consensus.cc:1261] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Refusing update from remote peer 6f553021ab414b35b8c0a38009352052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:40.353572 4237 raft_consensus.cc:1261] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Refusing update from remote peer 6f553021ab414b35b8c0a38009352052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:40.355108 4323 consensus_queue.cc:1035] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:40.355783 4319 consensus_queue.cc:1035] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:40.368700 4326 mvcc.cc:205] Tried to move back new op lower bound from 7315485001115709440 to 7315485000969637888. Current Snapshot: MvccSnapshot[applied={T|T < 7315485001115709440}]
I20260806 09:04:40.385766 3790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:40.386101 3790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:40.390208 3790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Sending DeleteTablet for 3 replicas of tablet 2844eec50383493da8428ea3c7452e9d
I20260806 09:04:40.391523 4216 tablet_service.cc:1514] Processing DeleteTablet for tablet 2844eec50383493da8428ea3c7452e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:40 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:41228
I20260806 09:04:40.391598 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet 2844eec50383493da8428ea3c7452e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:40 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:58466
I20260806 09:04:40.392781 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 2844eec50383493da8428ea3c7452e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:40 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:43976
I20260806 09:04:40.396255 3790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:40.397154 3790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:40.402969 4337 tablet_replica.cc:330] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: stopping tablet replica
I20260806 09:04:40.403821 4337 raft_consensus.cc:2229] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:40.404903 4337 raft_consensus.cc:2258] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:40.405820 4336 tablet_replica.cc:330] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: stopping tablet replica
I20260806 09:04:40.406381 4336 raft_consensus.cc:2229] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:40.407371 4336 raft_consensus.cc:2258] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:40.407482 4337 ts_tablet_manager.cc:1905] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:40.408680 4338 tablet_replica.cc:330] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: stopping tablet replica
I20260806 09:04:40.409174 4338 raft_consensus.cc:2229] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:40.409705 4338 raft_consensus.cc:2258] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:40.410300 3931 tablet_service.cc:1467] Processing CreateTablet for tablet a900a2caf94f4a19b980231b0d28c51d (DEFAULT_TABLE table=test-table [id=06244fede9384eef8e88dfecc871ad3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.411428 4074 tablet_service.cc:1467] Processing CreateTablet for tablet a900a2caf94f4a19b980231b0d28c51d (DEFAULT_TABLE table=test-table [id=06244fede9384eef8e88dfecc871ad3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.411777 4216 tablet_service.cc:1467] Processing CreateTablet for tablet a900a2caf94f4a19b980231b0d28c51d (DEFAULT_TABLE table=test-table [id=06244fede9384eef8e88dfecc871ad3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:40.412340 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a900a2caf94f4a19b980231b0d28c51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.412346 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a900a2caf94f4a19b980231b0d28c51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.411860 4336 ts_tablet_manager.cc:1905] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:40.420410 4312 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Bootstrap starting.
I20260806 09:04:40.420928 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a900a2caf94f4a19b980231b0d28c51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.422420 4312 tablet_bootstrap.cc:654] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.424789 4338 ts_tablet_manager.cc:1905] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:40.425853 4315 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Bootstrap starting.
I20260806 09:04:40.428038 4315 tablet_bootstrap.cc:654] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.437855 4336 ts_tablet_manager.cc:1918] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:40.438390 4336 log.cc:1196] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/2844eec50383493da8428ea3c7452e9d
I20260806 09:04:40.439527 4336 ts_tablet_manager.cc:1939] T 2844eec50383493da8428ea3c7452e9d P c248f66b886b4432b0d59b80d53643d1: Deleting consensus metadata
I20260806 09:04:40.442514 3776 catalog_manager.cc:4841] TS c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299): tablet 2844eec50383493da8428ea3c7452e9d (table test-table [id=54ac73bfcc284b67a3a84cb796557571]) successfully deleted
I20260806 09:04:40.446462 4313 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Bootstrap starting.
I20260806 09:04:40.447620 4312 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: No bootstrap required, opened a new log
I20260806 09:04:40.447829 4312 ts_tablet_manager.cc:1397] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.017s
I20260806 09:04:40.448658 4313 tablet_bootstrap.cc:654] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.448781 4312 raft_consensus.cc:348] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.449056 4312 raft_consensus.cc:374] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.449169 4312 raft_consensus.cc:729] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.449505 4312 consensus_queue.cc:260] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.449913 4315 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: No bootstrap required, opened a new log
I20260806 09:04:40.450105 4315 ts_tablet_manager.cc:1397] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.004s
I20260806 09:04:40.450809 4337 ts_tablet_manager.cc:1918] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:40.451387 4312 ts_tablet_manager.cc:1428] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 09:04:40.453351 4313 tablet_bootstrap.cc:492] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: No bootstrap required, opened a new log
I20260806 09:04:40.453521 4313 ts_tablet_manager.cc:1397] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 09:04:40.454046 4337 log.cc:1196] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/2844eec50383493da8428ea3c7452e9d
I20260806 09:04:40.454473 4313 raft_consensus.cc:348] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.454746 4313 raft_consensus.cc:374] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.454842 4337 ts_tablet_manager.cc:1939] T 2844eec50383493da8428ea3c7452e9d P 6f553021ab414b35b8c0a38009352052: Deleting consensus metadata
I20260806 09:04:40.454865 4313 raft_consensus.cc:729] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e84e1121920432ab107c553d4e669a8, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.455277 4313 consensus_queue.cc:260] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.456704 4313 ts_tablet_manager.cc:1428] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:40.451186 4315 raft_consensus.cc:348] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.457026 3775 catalog_manager.cc:4841] TS 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245): tablet 2844eec50383493da8428ea3c7452e9d (table test-table [id=54ac73bfcc284b67a3a84cb796557571]) successfully deleted
I20260806 09:04:40.457132 4315 raft_consensus.cc:374] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.457418 4315 raft_consensus.cc:729] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.457811 4315 consensus_queue.cc:260] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.459263 4338 ts_tablet_manager.cc:1918] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:40.459482 4338 log.cc:1196] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/2844eec50383493da8428ea3c7452e9d
I20260806 09:04:40.459620 4315 ts_tablet_manager.cc:1428] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260806 09:04:40.460173 4338 ts_tablet_manager.cc:1939] T 2844eec50383493da8428ea3c7452e9d P 2e84e1121920432ab107c553d4e669a8: Deleting consensus metadata
I20260806 09:04:40.462225 3775 catalog_manager.cc:4841] TS 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987): tablet 2844eec50383493da8428ea3c7452e9d (table test-table [id=54ac73bfcc284b67a3a84cb796557571]) successfully deleted
I20260806 09:04:40.553326 4318 raft_consensus.cc:482] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:40.553630 4318 raft_consensus.cc:504] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.555361 4318 leader_election.cc:288] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.563638 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.555806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:43101 (local address 127.27.187.129:33245)
0806 09:04:40.556016 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.556024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.556566 (+ 542us) server_negotiation.cc:407] Connection header received
0806 09:04:40.556921 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.556927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.557032 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.557111 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.557889 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.559024 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.561216 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.562539 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.562766 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.562844 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.563154 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.563307 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:04:40.563632 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.555792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33245 (local address 127.27.187.131:43101)
0806 09:04:40.556417 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.556433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.556709 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.557370 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.557374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.557402 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.557725 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.557739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.559167 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.559173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.561021 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.561042 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.562756 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.562764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.562963 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.563007 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.563120 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.563266 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":297,"threads_started":1}
I20260806 09:04:40.563917 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.556861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:54841 (local address 127.27.187.130:36987)
0806 09:04:40.557051 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.557056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.557628 (+ 572us) server_negotiation.cc:407] Connection header received
0806 09:04:40.557797 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.557803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.557886 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.557993 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:04:40.558612 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.559792 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.561848 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.562671 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.562849 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.562931 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.563463 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.563616 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:04:40.564610 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.556566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36987 (local address 127.27.187.131:54841)
0806 09:04:40.557498 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.557516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.557639 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.558036 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.558042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.558069 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:04:40.558413 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.558428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.559939 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.559945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.561697 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.561707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.563032 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.563039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.563251 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.563287 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.564141 (+ 854us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.564293 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":605,"threads_started":1}
I20260806 09:04:40.564950 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a900a2caf94f4a19b980231b0d28c51d" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052" is_pre_election: true
I20260806 09:04:40.565430 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a900a2caf94f4a19b980231b0d28c51d" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8" is_pre_election: true
I20260806 09:04:40.565459 3956 raft_consensus.cc:2454] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:40.565711 4100 raft_consensus.cc:2454] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:40.566197 4166 leader_election.cc:302] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f553021ab414b35b8c0a38009352052, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.566649 4318 raft_consensus.cc:2781] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:40.566807 4318 raft_consensus.cc:482] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:40.566913 4318 raft_consensus.cc:3037] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.570382 4318 raft_consensus.cc:504] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.571358 4318 leader_election.cc:288] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.572337 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a900a2caf94f4a19b980231b0d28c51d" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8"
I20260806 09:04:40.572359 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a900a2caf94f4a19b980231b0d28c51d" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052"
I20260806 09:04:40.572589 3956 raft_consensus.cc:3037] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.572584 4100 raft_consensus.cc:3037] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.576190 4100 raft_consensus.cc:2454] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:40.576220 3956 raft_consensus.cc:2454] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:40.576800 4166 leader_election.cc:302] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f553021ab414b35b8c0a38009352052, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.577420 4318 raft_consensus.cc:2781] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:40.578398 4318 raft_consensus.cc:686] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 1 LEADER]: Becoming Leader. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Running, Role: LEADER
I20260806 09:04:40.579917 4318 consensus_queue.cc:237] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.584189 3790 catalog_manager.cc:5495] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c248f66b886b4432b0d59b80d53643d1 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } health_report { overall_health: HEALTHY } } }
Aug 06 09:04:40 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.190@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:40.683382 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.649726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43253 (local address 127.0.0.1:38140)
0806 09:04:40.665641 (+ 15915us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.665663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.665848 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.666354 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.666359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.667543 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:40.667881 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.667894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.669683 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.669689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.671369 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.671386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.672415 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.672422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.672635 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.673525 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:40.673559 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:40.676846 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:40.680563 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:40.680580 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:40.680594 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:40.681015 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:40.681438 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:40.681445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:40.681449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:40.681596 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:40.682127 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:40.682135 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:40.682314 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.682483 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.682840 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15746,"thread_start_us":15638,"threads_started":1}
I20260806 09:04:40.683403 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.649768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.27.187.190:43253)
0806 09:04:40.665176 (+ 15408us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.665183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.665835 (+ 652us) server_negotiation.cc:407] Connection header received
0806 09:04:40.665961 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.665968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.666061 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.666322 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:40.668033 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.669523 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.671550 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.672203 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.672339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.677084 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:40.677089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:40.677092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:40.677130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:40.680401 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:40.681162 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:40.681167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:40.681177 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:40.681299 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:40.681720 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:40.681723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:40.681725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:40.681989 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:40.682173 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.682613 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.682967 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":15209,"thread_start_us":15942,"threads_started":1}
I20260806 09:04:40.690404 3790 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:38140:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:04:40.691869 3790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:40.700405 4074 tablet_service.cc:1467] Processing CreateTablet for tablet ef7437649233474c8bcbf77496800483 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee94fc544cda409ba38d168e88aab7f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:04:40.700525 4216 tablet_service.cc:1467] Processing CreateTablet for tablet ef7437649233474c8bcbf77496800483 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee94fc544cda409ba38d168e88aab7f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:04:40.701063 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7437649233474c8bcbf77496800483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.701063 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7437649233474c8bcbf77496800483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.703666 3931 tablet_service.cc:1467] Processing CreateTablet for tablet ef7437649233474c8bcbf77496800483 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee94fc544cda409ba38d168e88aab7f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:04:40.704360 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7437649233474c8bcbf77496800483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:40.710852 4312 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: Bootstrap starting.
I20260806 09:04:40.712704 4312 tablet_bootstrap.cc:654] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.713642 4313 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: Bootstrap starting.
I20260806 09:04:40.715184 4312 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: No bootstrap required, opened a new log
I20260806 09:04:40.715385 4312 ts_tablet_manager.cc:1397] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:04:40.715610 4313 tablet_bootstrap.cc:654] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.716146 4312 raft_consensus.cc:348] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.716409 4312 raft_consensus.cc:374] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.716499 4312 raft_consensus.cc:729] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.716801 4312 consensus_queue.cc:260] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.717295 4318 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.717617 4318 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.718205 4312 ts_tablet_manager.cc:1428] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:04:40.720592 4315 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: Bootstrap starting.
I20260806 09:04:40.722916 4315 tablet_bootstrap.cc:654] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:40.723869 4313 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: No bootstrap required, opened a new log
I20260806 09:04:40.724171 4313 ts_tablet_manager.cc:1397] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260806 09:04:40.725026 4313 raft_consensus.cc:348] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.725368 4313 raft_consensus.cc:374] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.725469 4313 raft_consensus.cc:729] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e84e1121920432ab107c553d4e669a8, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.725749 4313 consensus_queue.cc:260] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.726127 4315 tablet_bootstrap.cc:492] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: No bootstrap required, opened a new log
I20260806 09:04:40.726336 4315 ts_tablet_manager.cc:1397] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 09:04:40.726280 4320 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.726704 4320 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.727346 4315 raft_consensus.cc:348] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.727618 4315 raft_consensus.cc:374] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:40.727725 4315 raft_consensus.cc:729] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Initialized, Role: FOLLOWER
I20260806 09:04:40.728024 4315 consensus_queue.cc:260] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.728751 4313 ts_tablet_manager.cc:1428] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260806 09:04:40.728510 4331 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.728905 4331 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.729875 4315 ts_tablet_manager.cc:1428] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:04:40.857590 4318 raft_consensus.cc:482] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:40.857867 4318 raft_consensus.cc:504] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.858623 4318 leader_election.cc:288] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.859432 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7437649233474c8bcbf77496800483" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052" is_pre_election: true
I20260806 09:04:40.859555 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7437649233474c8bcbf77496800483" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8" is_pre_election: true
I20260806 09:04:40.859733 3956 raft_consensus.cc:2454] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:40.859870 4100 raft_consensus.cc:2454] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:40.860329 4166 leader_election.cc:302] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e84e1121920432ab107c553d4e669a8, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.860738 4318 raft_consensus.cc:2781] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:40.860869 4318 raft_consensus.cc:482] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:40.861011 4318 raft_consensus.cc:3037] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.863265 4318 raft_consensus.cc:504] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.863874 4318 leader_election.cc:288] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:40.864533 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7437649233474c8bcbf77496800483" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052"
I20260806 09:04:40.864533 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7437649233474c8bcbf77496800483" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8"
I20260806 09:04:40.864784 4100 raft_consensus.cc:3037] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.864799 3956 raft_consensus.cc:3037] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:40.867837 3956 raft_consensus.cc:2454] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:40.868057 4100 raft_consensus.cc:2454] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:40.868598 4166 leader_election.cc:302] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e84e1121920432ab107c553d4e669a8, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:40.869098 4318 raft_consensus.cc:2781] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:40.869483 4318 raft_consensus.cc:686] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [term 1 LEADER]: Becoming Leader. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Running, Role: LEADER
I20260806 09:04:40.869771 4318 consensus_queue.cc:237] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:40.871730 4344 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: TxnStatusTablet state changed. Reason: New leader c248f66b886b4432b0d59b80d53643d1. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.872068 4344 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:04:40.872680 4357 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:04:40.872925 4357 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:04:40.874356 3789 catalog_manager.cc:5495] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c248f66b886b4432b0d59b80d53643d1 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } health_report { overall_health: HEALTHY } } }
I20260806 09:04:40.923072 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.916112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.27.187.131:41299)
0806 09:04:40.916794 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.916799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.916815 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:04:40.916908 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.916912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.916999 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.917187 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:40.917640 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.918585 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.919989 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.920461 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.920606 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.921227 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:40.922108 (+ 881us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:40.922345 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.922432 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.922542 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":356,"threads_started":1}
I20260806 09:04:40.923043 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.915967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41299 (local address 127.0.0.1:41244)
0806 09:04:40.916230 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:40.916260 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:40.916425 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:40.917205 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:40.917210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:40.917223 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:04:40.917448 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.917467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.918757 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.918763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.919806 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:40.919819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.920636 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.920642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:40.920909 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.920976 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:04:40.922257 (+ 1281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:04:40.922304 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 09:04:40.922392 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 09:04:40.922610 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:04:40.928771 4100 raft_consensus.cc:1261] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Refusing update from remote peer c248f66b886b4432b0d59b80d53643d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:40.929062 3956 raft_consensus.cc:1261] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Refusing update from remote peer c248f66b886b4432b0d59b80d53643d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:40.929628 4344 consensus_queue.cc:1035] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:40.930104 4318 consensus_queue.cc:1035] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:40.952665 4320 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: TxnStatusTablet state changed. Reason: New leader c248f66b886b4432b0d59b80d53643d1. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.953042 4320 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.953114 4331 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: TxnStatusTablet state changed. Reason: New leader c248f66b886b4432b0d59b80d53643d1. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.953392 4331 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.955488 4361 mvcc.cc:205] Tried to move back new op lower bound from 7315485003475849216 to 7315485003245842432. Current Snapshot: MvccSnapshot[applied={T|T < 7315485003475849216}]
I20260806 09:04:40.959122 4362 mvcc.cc:205] Tried to move back new op lower bound from 7315485003475849216 to 7315485003245842432. Current Snapshot: MvccSnapshot[applied={T|T < 7315485003475849216}]
I20260806 09:04:40.959921 4344 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.960286 4344 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:04:40.964834 4320 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.965073 4320 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 2e84e1121920432ab107c553d4e669a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.971513 4344 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.971927 4344 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P c248f66b886b4432b0d59b80d53643d1: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:04:40.975988 4331 tablet_replica.cc:439] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } } }
I20260806 09:04:40.976302 4331 tablet_replica.cc:442] T ef7437649233474c8bcbf77496800483 P 6f553021ab414b35b8c0a38009352052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:04:40.983214 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:40.968705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.27.187.131:41299)
0806 09:04:40.968902 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:40.968912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:40.975460 (+ 6548us) server_negotiation.cc:407] Connection header received
0806 09:04:40.975862 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:40.975868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:40.975970 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:40.976038 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:40.977023 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.978393 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.980862 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:40.981532 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:40.981658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:40.982278 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:40.982436 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:40.982543 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:40.982777 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 09:04:40.982872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:04:40.984383 4366 consensus_queue.cc:1035] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:40.991878 4318 consensus_queue.cc:1035] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:41.015406 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.007583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:37981 (local address 127.27.187.131:41299)
0806 09:04:41.007845 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.007854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.008654 (+ 800us) server_negotiation.cc:407] Connection header received
0806 09:04:41.008903 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.008908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.009013 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.009123 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:41.009950 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.010897 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.013096 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.013622 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.013820 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.014354 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.014483 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.014600 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.014855 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.014937 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:04:41.015590 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.007624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41299 (local address 127.27.187.131:37981)
0806 09:04:41.008517 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:41.008536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:41.008750 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:41.009135 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:41.009140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:41.009152 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:04:41.009643 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.009654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.011029 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.011037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.012873 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.012893 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.013879 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.013893 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.014044 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.014115 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.014598 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.014623 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:04:41.014826 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 09:04:41.014997 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":438,"threads_started":1}
I20260806 09:04:41.087966 3789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.088312 3789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.092495 3789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Sending DeleteTablet for 3 replicas of tablet a900a2caf94f4a19b980231b0d28c51d
I20260806 09:04:41.093600 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet a900a2caf94f4a19b980231b0d28c51d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:58466
I20260806 09:04:41.093788 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet a900a2caf94f4a19b980231b0d28c51d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:43976
I20260806 09:04:41.094096 4216 tablet_service.cc:1514] Processing DeleteTablet for tablet a900a2caf94f4a19b980231b0d28c51d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:41228
I20260806 09:04:41.103909 4380 tablet_replica.cc:330] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: stopping tablet replica
I20260806 09:04:41.104453 4380 raft_consensus.cc:2229] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:41.105343 4380 raft_consensus.cc:2258] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.107125 4380 ts_tablet_manager.cc:1905] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.107867 4378 tablet_replica.cc:330] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: stopping tablet replica
I20260806 09:04:41.108238 4378 raft_consensus.cc:2229] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:41.109607 4379 tablet_replica.cc:330] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: stopping tablet replica
I20260806 09:04:41.109962 4379 raft_consensus.cc:2229] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:41.110271 4379 raft_consensus.cc:2258] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.111589 4379 ts_tablet_manager.cc:1905] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.114984 4378 raft_consensus.cc:2258] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.116174 4378 ts_tablet_manager.cc:1905] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.122028 4379 ts_tablet_manager.cc:1918] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.122323 4379 log.cc:1196] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/a900a2caf94f4a19b980231b0d28c51d
I20260806 09:04:41.123108 4379 ts_tablet_manager.cc:1939] T a900a2caf94f4a19b980231b0d28c51d P 2e84e1121920432ab107c553d4e669a8: Deleting consensus metadata
I20260806 09:04:41.124696 3775 catalog_manager.cc:4841] TS 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987): tablet a900a2caf94f4a19b980231b0d28c51d (table test-table [id=06244fede9384eef8e88dfecc871ad3e]) successfully deleted
I20260806 09:04:41.127164 4380 ts_tablet_manager.cc:1918] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.127489 4380 log.cc:1196] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/a900a2caf94f4a19b980231b0d28c51d
I20260806 09:04:41.127717 4378 ts_tablet_manager.cc:1918] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.127889 4378 log.cc:1196] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/a900a2caf94f4a19b980231b0d28c51d
I20260806 09:04:41.128311 4380 ts_tablet_manager.cc:1939] T a900a2caf94f4a19b980231b0d28c51d P c248f66b886b4432b0d59b80d53643d1: Deleting consensus metadata
I20260806 09:04:41.128439 4378 ts_tablet_manager.cc:1939] T a900a2caf94f4a19b980231b0d28c51d P 6f553021ab414b35b8c0a38009352052: Deleting consensus metadata
I20260806 09:04:41.129799 3775 catalog_manager.cc:4841] TS 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245): tablet a900a2caf94f4a19b980231b0d28c51d (table test-table [id=06244fede9384eef8e88dfecc871ad3e]) successfully deleted
I20260806 09:04:41.129799 3776 catalog_manager.cc:4841] TS c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299): tablet a900a2caf94f4a19b980231b0d28c51d (table test-table [id=06244fede9384eef8e88dfecc871ad3e]) successfully deleted
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:04:41.181087 4351 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:41.168109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.27.187.190:43253)
0806 09:04:41.168393 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.168403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.168998 (+ 595us) server_negotiation.cc:407] Connection header received
0806 09:04:41.169281 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.169287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.169376 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.169503 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:41.171218 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.172464 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.173983 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.174639 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.174806 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.180424 (+ 5618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM
I20260806 09:04:41.199669 4351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.183743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.27.187.190:43253)
0806 09:04:41.183995 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.184002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.184315 (+ 313us) server_negotiation.cc:407] Connection header received
0806 09:04:41.184566 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.184571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.184680 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.184805 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:41.186329 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.187197 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.188506 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.188971 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.189117 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.193348 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:41.193354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:41.193357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:41.193399 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:41.196662 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.197491 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.197497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.197501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.197660 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.198088 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.198093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.198096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.198406 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:41.198720 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.199032 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.199303 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:04:41.204017 3789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:04:41.204741 3789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:41.213115 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 025dfffa277a4551a872615931dcbc05 (DEFAULT_TABLE table=test-table [id=d6bc081ad6df4b9b956314de9301f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.213290 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 025dfffa277a4551a872615931dcbc05 (DEFAULT_TABLE table=test-table [id=d6bc081ad6df4b9b956314de9301f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.213711 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025dfffa277a4551a872615931dcbc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.213739 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025dfffa277a4551a872615931dcbc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.219058 4313 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Bootstrap starting.
I20260806 09:04:41.220103 4216 tablet_service.cc:1467] Processing CreateTablet for tablet 025dfffa277a4551a872615931dcbc05 (DEFAULT_TABLE table=test-table [id=d6bc081ad6df4b9b956314de9301f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.220602 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025dfffa277a4551a872615931dcbc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.220745 4315 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Bootstrap starting.
I20260806 09:04:41.220960 4313 tablet_bootstrap.cc:654] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.222484 4315 tablet_bootstrap.cc:654] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.231973 4313 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: No bootstrap required, opened a new log
I20260806 09:04:41.232199 4313 ts_tablet_manager.cc:1397] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260806 09:04:41.233067 4313 raft_consensus.cc:348] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.233371 4313 raft_consensus.cc:374] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.233460 4313 raft_consensus.cc:729] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e84e1121920432ab107c553d4e669a8, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.233717 4313 consensus_queue.cc:260] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.233987 4315 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: No bootstrap required, opened a new log
I20260806 09:04:41.234155 4315 ts_tablet_manager.cc:1397] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260806 09:04:41.234333 4313 ts_tablet_manager.cc:1428] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:04:41.235055 4315 raft_consensus.cc:348] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.235425 4315 raft_consensus.cc:374] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.235546 4315 raft_consensus.cc:729] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.237351 4315 consensus_queue.cc:260] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.237777 4395 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Bootstrap starting.
I20260806 09:04:41.240218 4315 ts_tablet_manager.cc:1428] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260806 09:04:41.240342 4395 tablet_bootstrap.cc:654] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.243691 4395 tablet_bootstrap.cc:492] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: No bootstrap required, opened a new log
I20260806 09:04:41.243877 4395 ts_tablet_manager.cc:1397] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 09:04:41.244637 4395 raft_consensus.cc:348] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.244843 4395 raft_consensus.cc:374] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.244961 4395 raft_consensus.cc:729] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.245213 4395 consensus_queue.cc:260] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.245837 4395 ts_tablet_manager.cc:1428] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:04:41.326203 4318 raft_consensus.cc:482] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:41.326555 4318 raft_consensus.cc:504] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.327459 4318 leader_election.cc:288] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:41.328131 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025dfffa277a4551a872615931dcbc05" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8" is_pre_election: true
I20260806 09:04:41.328150 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025dfffa277a4551a872615931dcbc05" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052" is_pre_election: true
I20260806 09:04:41.328429 4099 raft_consensus.cc:2454] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:41.328444 3956 raft_consensus.cc:2454] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 0.
I20260806 09:04:41.328902 4166 leader_election.cc:302] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f553021ab414b35b8c0a38009352052, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:41.329231 4318 raft_consensus.cc:2781] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:41.329380 4318 raft_consensus.cc:482] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:41.329483 4318 raft_consensus.cc:3037] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.332139 4318 raft_consensus.cc:504] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.332827 4318 leader_election.cc:288] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Requested vote from peers 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245), 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987)
I20260806 09:04:41.333498 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025dfffa277a4551a872615931dcbc05" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8"
I20260806 09:04:41.333503 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025dfffa277a4551a872615931dcbc05" candidate_uuid: "c248f66b886b4432b0d59b80d53643d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f553021ab414b35b8c0a38009352052"
I20260806 09:04:41.333736 4099 raft_consensus.cc:3037] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.333766 3956 raft_consensus.cc:3037] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.336899 3956 raft_consensus.cc:2454] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:41.336913 4099 raft_consensus.cc:2454] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c248f66b886b4432b0d59b80d53643d1 in term 1.
I20260806 09:04:41.337471 4166 leader_election.cc:302] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e84e1121920432ab107c553d4e669a8, c248f66b886b4432b0d59b80d53643d1; no voters:
I20260806 09:04:41.337877 4318 raft_consensus.cc:2781] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:41.338078 4318 raft_consensus.cc:686] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 1 LEADER]: Becoming Leader. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Running, Role: LEADER
I20260806 09:04:41.338389 4318 consensus_queue.cc:237] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.341856 3789 catalog_manager.cc:5495] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c248f66b886b4432b0d59b80d53643d1 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "c248f66b886b4432b0d59b80d53643d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } health_report { overall_health: HEALTHY } } }
I20260806 09:04:41.365001 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.359018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.27.187.131:41299)
0806 09:04:41.359220 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.359227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.359403 (+ 176us) server_negotiation.cc:407] Connection header received
0806 09:04:41.359592 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.359598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.359687 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.359792 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:41.360152 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.361343 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.362827 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.363515 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.363650 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.364109 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.364254 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.364341 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.364506 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.364642 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:04:41.368165 4099 raft_consensus.cc:1261] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Refusing update from remote peer c248f66b886b4432b0d59b80d53643d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:41.368258 3956 raft_consensus.cc:1261] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Refusing update from remote peer c248f66b886b4432b0d59b80d53643d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:41.369045 4318 consensus_queue.cc:1035] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:41.369427 4344 consensus_queue.cc:1035] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:41.384663 3791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.384974 3791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.388401 3791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Sending DeleteTablet for 3 replicas of tablet 025dfffa277a4551a872615931dcbc05
I20260806 09:04:41.389509 4216 tablet_service.cc:1514] Processing DeleteTablet for tablet 025dfffa277a4551a872615931dcbc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:41228
I20260806 09:04:41.389722 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 025dfffa277a4551a872615931dcbc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:43976
I20260806 09:04:41.390056 4380 tablet_replica.cc:330] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: stopping tablet replica
I20260806 09:04:41.390280 4380 raft_consensus.cc:2229] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:41.390646 4380 raft_consensus.cc:2258] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.390532 3791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 09:04:41.390998 4379 tablet_replica.cc:330] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: stopping tablet replica
I20260806 09:04:41.391197 4379 raft_consensus.cc:2229] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 09:04:41.391366 3791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:04:41.391436 4379 raft_consensus.cc:2258] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.392443 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet 025dfffa277a4551a872615931dcbc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:58466
I20260806 09:04:41.392647 4379 ts_tablet_manager.cc:1905] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.392733 4378 tablet_replica.cc:330] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: stopping tablet replica
I20260806 09:04:41.392925 4378 raft_consensus.cc:2229] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:41.393193 4378 raft_consensus.cc:2258] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.394356 4378 ts_tablet_manager.cc:1905] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.396281 4380 ts_tablet_manager.cc:1905] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.401290 4378 ts_tablet_manager.cc:1918] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:41.401470 4378 log.cc:1196] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/025dfffa277a4551a872615931dcbc05
I20260806 09:04:41.402235 4216 tablet_service.cc:1467] Processing CreateTablet for tablet 83d371368d8646e98eaa16b659efe804 (DEFAULT_TABLE table=test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.402590 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 83d371368d8646e98eaa16b659efe804 (DEFAULT_TABLE table=test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.402768 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d371368d8646e98eaa16b659efe804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.403061 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d371368d8646e98eaa16b659efe804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.403676 4379 ts_tablet_manager.cc:1918] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:41.403828 4379 log.cc:1196] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/025dfffa277a4551a872615931dcbc05
I20260806 09:04:41.404299 4379 ts_tablet_manager.cc:1939] T 025dfffa277a4551a872615931dcbc05 P 2e84e1121920432ab107c553d4e669a8: Deleting consensus metadata
I20260806 09:04:41.407223 3775 catalog_manager.cc:4841] TS 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987): tablet 025dfffa277a4551a872615931dcbc05 (table test-table [id=d6bc081ad6df4b9b956314de9301f049]) successfully deleted
I20260806 09:04:41.408203 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 83d371368d8646e98eaa16b659efe804 (DEFAULT_TABLE table=test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:04:41.408998 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d371368d8646e98eaa16b659efe804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:04:41.409966 4395 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Bootstrap starting.
I20260806 09:04:41.410473 4378 ts_tablet_manager.cc:1939] T 025dfffa277a4551a872615931dcbc05 P 6f553021ab414b35b8c0a38009352052: Deleting consensus metadata
I20260806 09:04:41.411895 4395 tablet_bootstrap.cc:654] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.411927 3775 catalog_manager.cc:4841] TS 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245): tablet 025dfffa277a4551a872615931dcbc05 (table test-table [id=d6bc081ad6df4b9b956314de9301f049]) successfully deleted
I20260806 09:04:41.414618 4395 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: No bootstrap required, opened a new log
I20260806 09:04:41.414856 4395 ts_tablet_manager.cc:1397] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 09:04:41.416736 4313 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Bootstrap starting.
I20260806 09:04:41.417466 4395 raft_consensus.cc:348] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.417780 4395 raft_consensus.cc:374] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.417858 4395 raft_consensus.cc:729] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c248f66b886b4432b0d59b80d53643d1, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.418181 4395 consensus_queue.cc:260] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.418599 4313 tablet_bootstrap.cc:654] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.418813 4395 ts_tablet_manager.cc:1428] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260806 09:04:41.420277 4315 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Bootstrap starting.
I20260806 09:04:41.422073 4315 tablet_bootstrap.cc:654] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:41.424443 4380 ts_tablet_manager.cc:1918] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:04:41.424651 4380 log.cc:1196] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/025dfffa277a4551a872615931dcbc05
I20260806 09:04:41.427147 4380 ts_tablet_manager.cc:1939] T 025dfffa277a4551a872615931dcbc05 P c248f66b886b4432b0d59b80d53643d1: Deleting consensus metadata
I20260806 09:04:41.427631 4313 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: No bootstrap required, opened a new log
I20260806 09:04:41.427827 4313 ts_tablet_manager.cc:1397] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 09:04:41.428671 3776 catalog_manager.cc:4841] TS c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299): tablet 025dfffa277a4551a872615931dcbc05 (table test-table [id=d6bc081ad6df4b9b956314de9301f049]) successfully deleted
I20260806 09:04:41.428716 4313 raft_consensus.cc:348] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.429005 4313 raft_consensus.cc:374] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.429097 4313 raft_consensus.cc:729] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e84e1121920432ab107c553d4e669a8, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.429358 4313 consensus_queue.cc:260] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.429960 4313 ts_tablet_manager.cc:1428] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260806 09:04:41.431741 4315 tablet_bootstrap.cc:492] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: No bootstrap required, opened a new log
I20260806 09:04:41.431938 4315 ts_tablet_manager.cc:1397] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260806 09:04:41.432786 4315 raft_consensus.cc:348] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.433079 4315 raft_consensus.cc:374] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:41.433171 4315 raft_consensus.cc:729] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Initialized, Role: FOLLOWER
I20260806 09:04:41.433422 4315 consensus_queue.cc:260] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.434000 4315 ts_tablet_manager.cc:1428] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:04:41.463084 4331 raft_consensus.cc:482] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:04:41.463337 4331 raft_consensus.cc:504] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.464181 4331 leader_election.cc:288] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987), c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299)
I20260806 09:04:41.464802 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d371368d8646e98eaa16b659efe804" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8" is_pre_election: true
I20260806 09:04:41.464835 4237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d371368d8646e98eaa16b659efe804" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c248f66b886b4432b0d59b80d53643d1" is_pre_election: true
I20260806 09:04:41.465063 4099 raft_consensus.cc:2454] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 0.
I20260806 09:04:41.465107 4237 raft_consensus.cc:2454] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 0.
I20260806 09:04:41.465616 3881 leader_election.cc:302] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e84e1121920432ab107c553d4e669a8, 6f553021ab414b35b8c0a38009352052; no voters:
I20260806 09:04:41.466110 4331 raft_consensus.cc:2781] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:04:41.466285 4331 raft_consensus.cc:482] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:04:41.466440 4331 raft_consensus.cc:3037] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.469273 4331 raft_consensus.cc:504] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.469976 4331 leader_election.cc:288] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 election: Requested vote from peers 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987), c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299)
I20260806 09:04:41.470757 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d371368d8646e98eaa16b659efe804" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e84e1121920432ab107c553d4e669a8"
I20260806 09:04:41.470790 4237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d371368d8646e98eaa16b659efe804" candidate_uuid: "6f553021ab414b35b8c0a38009352052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c248f66b886b4432b0d59b80d53643d1"
I20260806 09:04:41.470963 4099 raft_consensus.cc:3037] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.471033 4237 raft_consensus.cc:3037] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:41.473467 4099 raft_consensus.cc:2454] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 1.
I20260806 09:04:41.474686 4237 raft_consensus.cc:2454] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f553021ab414b35b8c0a38009352052 in term 1.
I20260806 09:04:41.474778 3881 leader_election.cc:302] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e84e1121920432ab107c553d4e669a8, 6f553021ab414b35b8c0a38009352052; no voters:
I20260806 09:04:41.475117 4331 raft_consensus.cc:2781] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:41.476166 4331 raft_consensus.cc:686] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 1 LEADER]: Becoming Leader. State: Replica: 6f553021ab414b35b8c0a38009352052, State: Running, Role: LEADER
I20260806 09:04:41.476495 4331 consensus_queue.cc:237] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } }
I20260806 09:04:41.479509 3791 catalog_manager.cc:5495] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f553021ab414b35b8c0a38009352052 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "6f553021ab414b35b8c0a38009352052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f553021ab414b35b8c0a38009352052" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 33245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 } health_report { overall_health: UNKNOWN } } }
I20260806 09:04:41.511045 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.504086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.27.187.129:33245)
0806 09:04:41.504757 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.504762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.504779 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:04:41.504836 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.504839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.504900 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.504985 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:41.505528 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.506396 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.508748 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.509266 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.509378 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.510025 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.510150 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.510242 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.510467 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.510551 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":414,"threads_started":1}
I20260806 09:04:41.530519 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.522736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34531 (local address 127.27.187.131:41299)
0806 09:04:41.522954 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.522968 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.523771 (+ 803us) server_negotiation.cc:407] Connection header received
0806 09:04:41.524036 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.524041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.524175 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.524260 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:41.524912 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.526012 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.528420 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.528943 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.529065 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.529706 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.529846 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.529946 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.530152 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.530236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:04:41.530725 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.522761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41299 (local address 127.27.187.129:34531)
0806 09:04:41.523541 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:41.523565 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:41.523864 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:41.524316 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:41.524323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:41.524341 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:04:41.524666 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.524681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.526263 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.526270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.528218 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.528241 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.529163 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.529170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.529381 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.529481 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.529977 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.529991 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 09:04:41.530081 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 09:04:41.530262 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":373,"threads_started":1}
I20260806 09:04:41.540766 4237 raft_consensus.cc:1261] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Refusing update from remote peer 6f553021ab414b35b8c0a38009352052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:41.540959 4099 raft_consensus.cc:1261] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Refusing update from remote peer 6f553021ab414b35b8c0a38009352052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:04:41.541527 4331 consensus_queue.cc:1035] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c248f66b886b4432b0d59b80d53643d1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:41.541868 4404 consensus_queue.cc:1035] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e84e1121920432ab107c553d4e669a8" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:04:41.583621 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.574444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:41643 (local address 127.27.187.129:33245)
0806 09:04:41.574697 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.574705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.575931 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 09:04:41.576255 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.576262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.576374 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.576505 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:04:41.577043 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.578371 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.580919 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.582052 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.582260 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.582932 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.583054 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.583138 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.583316 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.583392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:04:41.584148 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.573908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:33245 (local address 127.27.187.131:41643)
0806 09:04:41.575825 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:41.575847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:41.576009 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:41.576528 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:41.576532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:41.576544 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:04:41.576858 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.576867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.578551 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.578558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.580541 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:41.580563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.582422 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.582430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:41.582637 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.582733 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:04:41.583176 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:04:41.583189 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 09:04:41.583272 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 09:04:41.583462 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":1693,"threads_started":1}
I20260806 09:04:41.620844 3791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.621210 3791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:04:41.624902 3791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb71f3ba0db40f997bd30d2b3dc9ff1: Sending DeleteTablet for 3 replicas of tablet 83d371368d8646e98eaa16b659efe804
I20260806 09:04:41.626101 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d371368d8646e98eaa16b659efe804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:43976
I20260806 09:04:41.626317 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d371368d8646e98eaa16b659efe804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:58466
I20260806 09:04:41.626721 4378 tablet_replica.cc:330] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: stopping tablet replica
I20260806 09:04:41.626744 4379 tablet_replica.cc:330] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: stopping tablet replica
I20260806 09:04:41.626905 4378 raft_consensus.cc:2229] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:04:41.626959 4379 raft_consensus.cc:2229] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:41.627208 4379 raft_consensus.cc:2258] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.627207 4378 raft_consensus.cc:2258] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.627648 4216 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d371368d8646e98eaa16b659efe804 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:04:41 UTC) from {username='oryx', principal='oryx/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:41228
I20260806 09:04:41.627979 4380 tablet_replica.cc:330] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: stopping tablet replica
I20260806 09:04:41.628196 4380 raft_consensus.cc:2229] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:04:41.628444 4380 raft_consensus.cc:2258] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:04:41.628608 4378 ts_tablet_manager.cc:1905] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.628759 4379 ts_tablet_manager.cc:1905] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.632319 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 3864
I20260806 09:04:41.636476 4379 ts_tablet_manager.cc:1918] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.636639 4379 log.cc:1196] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/83d371368d8646e98eaa16b659efe804
I20260806 09:04:41.637176 4379 ts_tablet_manager.cc:1939] T 83d371368d8646e98eaa16b659efe804 P 2e84e1121920432ab107c553d4e669a8: Deleting consensus metadata
I20260806 09:04:41.638501 4378 ts_tablet_manager.cc:1918] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.638556 3775 catalog_manager.cc:4841] TS 2e84e1121920432ab107c553d4e669a8 (127.27.187.130:36987): tablet 83d371368d8646e98eaa16b659efe804 (table test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]) successfully deleted
I20260806 09:04:41.638701 4378 log.cc:1196] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/83d371368d8646e98eaa16b659efe804
I20260806 09:04:41.639310 4378 ts_tablet_manager.cc:1939] T 83d371368d8646e98eaa16b659efe804 P 6f553021ab414b35b8c0a38009352052: Deleting consensus metadata
I20260806 09:04:41.640370 4380 ts_tablet_manager.cc:1905] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:04:41.643260 3775 catalog_manager.cc:4841] TS 6f553021ab414b35b8c0a38009352052 (127.27.187.129:33245): tablet 83d371368d8646e98eaa16b659efe804 (table test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]) successfully deleted
I20260806 09:04:41.648443 4380 ts_tablet_manager.cc:1918] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:04:41.648748 4380 log.cc:1196] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/83d371368d8646e98eaa16b659efe804
I20260806 09:04:41.649369 4380 ts_tablet_manager.cc:1939] T 83d371368d8646e98eaa16b659efe804 P c248f66b886b4432b0d59b80d53643d1: Deleting consensus metadata
I20260806 09:04:41.650866 3776 catalog_manager.cc:4841] TS c248f66b886b4432b0d59b80d53643d1 (127.27.187.131:41299): tablet 83d371368d8646e98eaa16b659efe804 (table test-table [id=1f516545f2bf4bb0b0c8ff4116839d17]) successfully deleted
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.129@KRBTEST.COM
I20260806 09:04:41.665295 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.633781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.27.187.129:33245)
0806 09:04:41.633995 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.634002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.643975 (+ 9973us) server_negotiation.cc:407] Connection header received
0806 09:04:41.644479 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.644485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.644583 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.644656 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:41.646521 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.647731 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.649613 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.650344 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.650461 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.656537 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:41.656559 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:41.656581 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:41.656633 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:41.661038 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.662281 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.662292 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.662296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.662484 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.663058 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.663064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.663067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.663614 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:41.663938 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.664354 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.664606 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:04:41.666308 3999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:41.809510 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 3864
I20260806 09:04:41.832504 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4008
Aug 06 09:04:41 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.130@KRBTEST.COM
I20260806 09:04:41.854883 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:41.833405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.27.187.130:36987)
0806 09:04:41.833955 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:41.833960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:41.833977 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:04:41.834032 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:41.834036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:41.834104 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:41.834202 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:41.836187 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.837091 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.839403 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:41.839961 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:41.840093 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:41.846360 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:41.846379 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:41.846417 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:41.846467 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:41.850763 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.852027 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.852036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.852050 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.852302 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:41.852923 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:41.852928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:41.852931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:41.853444 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:41.853731 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:41.854244 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 09:04:41.854477 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
I20260806 09:04:41.855649 4140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:41.995972 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4008
I20260806 09:04:42.016889 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4149
Aug 06 09:04:42 dist-test-slave-8tdl krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.187.131@KRBTEST.COM
I20260806 09:04:42.034572 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:42.017888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.27.187.131:41299)
0806 09:04:42.018088 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:42.018096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:42.018118 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:04:42.018194 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:42.018198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:42.018286 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:42.018597 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:42.020049 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:42.020859 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:42.022306 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:42.022879 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:42.023024 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:42.027860 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:42.027878 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:42.027895 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:42.027938 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:42.031458 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:42.032319 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:42.032326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:42.032329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:42.032479 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:42.032885 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:42.032888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:42.032890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:42.033306 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:42.033554 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:42.033874 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:04:42.034069 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:04:42.035398 4282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:42.193456 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:42.191375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36987 (local address 127.27.187.131:46625)
0806 09:04:42.193085 (+ 1710us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:42.193249 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.130:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":951,"threads_started":1}
I20260806 09:04:42.193529 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4149
I20260806 09:04:42.222453 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 3758
I20260806 09:04:42.223935 3831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:42.345764 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 3758
2026-08-06T09:04:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6416 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:42 dist-test-slave-8tdl krb5kdc[4426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:42 dist-test-slave-8tdl krb5kdc[4426](info): set up 2 sockets
Aug 06 09:04:42 dist-test-slave-8tdl krb5kdc[4426](info): commencing operation
krb5kdc: starting...
W20260806 09:04:43.494930 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:43 dist-test-slave-8tdl krb5kdc[4426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:04:43.817150 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38951
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:44.213162 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:44.213495 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:44.213706 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:44.221633 4439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:44.221764 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:44.221856 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:44.221947 4439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:44.222023 4439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:44.232630 4439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:44.234120 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:44.235560 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:44.249199 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:44.249305 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:44.249292 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:44.250046 4439 server_base.cc:1034] running on GCE node
I20260806 09:04:44.251098 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:44.252820 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:44.254097 4439 hybrid_clock.cc:648] HybridClock initialized: now 1786007084254038 us; error 84 us; skew 500 ppm
I20260806 09:04:44.254509 4439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T09:04:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1985 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:04:44 dist-test-slave-8tdl krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:04:44 dist-test-slave-8tdl krb5kdc[4453](info): set up 2 sockets
Aug 06 09:04:44 dist-test-slave-8tdl krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260806 09:04:45.519835 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.005s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:04:45 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:04:45.830545 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45697
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:46.196134 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:46.196480 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:46.196667 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:46.204407 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:46.204550 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:46.204645 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:46.204744 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:46.204851 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:46.215530 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:46.217042 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:46.218473 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:46.231601 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:46.232098 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:46.231832 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:46.233160 4466 server_base.cc:1034] running on GCE node
I20260806 09:04:46.234032 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:46.235780 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:46.237048 4466 hybrid_clock.cc:648] HybridClock initialized: now 1786007086236974 us; error 65 us; skew 500 ppm
I20260806 09:04:46.237540 4466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:46 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007086, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:46.241214 4466 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:04:46.243000 4466 webserver.cc:460] Webserver started at http://127.27.187.190:38225/ using document root <none> and password file <none>
I20260806 09:04:46.243732 4466 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:46.243837 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:46.244110 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:46.247371 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "1ae65e32d926439ab62188d62a51b2a6"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "41c8167f28b6e990f55e5881ac8b5e74"
server_key_iv: "91b01b3aa2af9b517e63e59a81bb35ab"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.248160 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "1ae65e32d926439ab62188d62a51b2a6"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "41c8167f28b6e990f55e5881ac8b5e74"
server_key_iv: "91b01b3aa2af9b517e63e59a81bb35ab"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.254357 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:04:46.258244 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:46.259929 4466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:46.260102 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "1ae65e32d926439ab62188d62a51b2a6"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "41c8167f28b6e990f55e5881ac8b5e74"
server_key_iv: "91b01b3aa2af9b517e63e59a81bb35ab"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.260386 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:46.282759 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:46.288434 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:46.288796 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:46.310396 4466 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45697
I20260806 09:04:46.310407 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45697 every 8 connection(s)
I20260806 09:04:46.311961 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:46.314028 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4466
I20260806 09:04:46.314294 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:04:46.315215 28398 external_mini_cluster.cc:1428] Setting key 6be23c55029cc3badf7472ab86a1745e
I20260806 09:04:46.317332 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:04:46 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:46.332551 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Bootstrap starting.
I20260806 09:04:46.336290 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Neither blocks nor log segments found. Creating new log.
I20260806 09:04:46.337432 4534 log.cc:824] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:46.340389 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:46.317534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.27.187.190:45697)
0806 09:04:46.318401 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:46.318413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:46.318458 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:04:46.320267 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:46.320290 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:46.320715 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:46.321093 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:46.323267 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:46.324722 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:46.326577 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:46.327384 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:46.327534 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:46.332537 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:46.332546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:46.332564 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:46.332612 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:46.335922 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:46.336803 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:46.336811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:46.336818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:46.336997 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:46.337418 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:46.337425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:46.337433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:46.338063 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:46.338406 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:46.338934 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 09:04:46.339375 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":414,"threads_started":1}
I20260806 09:04:46.340715 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: No bootstrap required, opened a new log
I20260806 09:04:46.349411 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:46.349917 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:04:46.350039 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae65e32d926439ab62188d62a51b2a6, State: Initialized, Role: FOLLOWER
I20260806 09:04:46.350831 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:46.351114 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:46.351323 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:46.351509 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:04:46.353847 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:46.354426 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae65e32d926439ab62188d62a51b2a6; no voters:
I20260806 09:04:46.355134 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:04:46.355336 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:04:46.356510 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae65e32d926439ab62188d62a51b2a6, State: Running, Role: LEADER
I20260806 09:04:46.357159 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:46.357470 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:46.361215 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ae65e32d926439ab62188d62a51b2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } } }
I20260806 09:04:46.361198 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae65e32d926439ab62188d62a51b2a6. Latest consensus state: current_term: 1 leader_uuid: "1ae65e32d926439ab62188d62a51b2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } } }
I20260806 09:04:46.361719 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:46.361719 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:46.368220 4547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:46.375335 4547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:46.385548 4547 catalog_manager.cc:1347] Generated new cluster ID: fb85bc0c815f415dbed6b8ff66a4e2f3
I20260806 09:04:46.385720 4547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:46.407104 4547 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:04:46.409197 4547 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:46.418222 4547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Generated new TSK 0
I20260806 09:04:46.419471 4547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:04:46.491676 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:46.856446 4560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:46.856799 4560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:46.856961 4560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:46.865156 4560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:46.865294 4560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:46.865468 4560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:46.876014 4560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:46.877455 4560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:46.878913 4560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:46.893347 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:46.893406 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:46.893494 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:46.894834 4560 server_base.cc:1034] running on GCE node
I20260806 09:04:46.895475 4560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:46.896476 4560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:46.897753 4560 hybrid_clock.cc:648] HybridClock initialized: now 1786007086897680 us; error 57 us; skew 500 ppm
I20260806 09:04:46.898141 4560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:46 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007086, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:46.901710 4560 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:04:46.903421 4560 webserver.cc:460] Webserver started at http://127.27.187.129:35773/ using document root <none> and password file <none>
I20260806 09:04:46.904204 4560 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:46.904353 4560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:46.904739 4560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:46.907951 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "67b96efea5174a2ab2e7270e377856c3"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "e99484bd5b512001ea6ac360880a577a"
server_key_iv: "f53be893ae6136d0e24c269b556a77b8"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.908845 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "67b96efea5174a2ab2e7270e377856c3"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "e99484bd5b512001ea6ac360880a577a"
server_key_iv: "f53be893ae6136d0e24c269b556a77b8"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.914516 4560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:04:46.918509 4575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:46.920228 4560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:46.920446 4560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "67b96efea5174a2ab2e7270e377856c3"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "e99484bd5b512001ea6ac360880a577a"
server_key_iv: "f53be893ae6136d0e24c269b556a77b8"
server_key_version: "encryptionkey@0"
I20260806 09:04:46.920686 4560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:46.944646 4560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:46.949654 4560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:46.950003 4560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:46.951354 4560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:46.953091 4560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:46.953192 4560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:46.953328 4560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:46.953415 4560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:46.990228 4560 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:39861
I20260806 09:04:46.990366 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:39861 every 8 connection(s)
I20260806 09:04:46.991675 4560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:46.992409 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4560
I20260806 09:04:46.992621 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:04:46.993515 28398 external_mini_cluster.cc:1428] Setting key c3beae97717b0a2bc040e94aa2207d50
Aug 06 09:04:47 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007086, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:47.019151 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:46.994955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:53057 (local address 127.27.187.190:45697)
0806 09:04:46.995647 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:46.995654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:46.996787 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 09:04:47.000102 (+ 3315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:47.000108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:47.000209 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:47.000338 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:47.003042 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.004039 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.006840 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.007320 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.007445 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:47.012163 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:47.012170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:47.012174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:47.012215 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:47.015177 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:47.016418 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:47.016423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:47.016426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:47.016571 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:47.017126 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:47.017131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:47.017133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:47.017432 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:47.017547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:47.018597 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0806 09:04:47.018880 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":507,"threads_started":1}
I20260806 09:04:47.020813 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:46.995488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:53057)
0806 09:04:46.996569 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:46.996635 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:46.999640 (+ 3005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:47.000680 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:47.000697 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:47.001693 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:47.002722 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:47.002763 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.004297 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.004305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:47.006610 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:47.006632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.007598 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.007607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:47.007980 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:47.008832 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:47.008857 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:47.011782 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:47.015366 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:47.015379 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:47.015404 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:47.016229 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:47.016698 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:47.016706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:47.016712 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:47.016990 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:47.017575 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:47.017586 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:47.018121 (+ 535us) client_negotiation.cc:769] Sending connection context
0806 09:04:47.018481 (+ 360us) client_negotiation.cc:240] Negotiation successful
0806 09:04:47.019203 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":396,"threads_started":1}
I20260806 09:04:47.022778 4691 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45697
I20260806 09:04:47.023129 4691 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:47.023931 4691 heartbeater.cc:511] Master 127.27.187.190:45697 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:04:47.026793 4498 ts_manager.cc:194] Registered new tserver with Master: 67b96efea5174a2ab2e7270e377856c3 (127.27.187.129:39861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:04:47.028579 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:53057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:04:47.061812 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:47.458665 4697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:47.459076 4697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:47.459249 4697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:47.467422 4697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:47.467604 4697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:47.467801 4697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:47.479312 4697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:47.480821 4697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:47.482393 4697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:47.497298 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:47.497270 4706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:47.497323 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:47.500499 4697 server_base.cc:1034] running on GCE node
I20260806 09:04:47.501274 4697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:47.502313 4697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:47.503580 4697 hybrid_clock.cc:648] HybridClock initialized: now 1786007087503546 us; error 47 us; skew 500 ppm
I20260806 09:04:47.503939 4697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:47 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007087, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:47.507565 4697 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:04:47.509439 4697 webserver.cc:460] Webserver started at http://127.27.187.130:41117/ using document root <none> and password file <none>
I20260806 09:04:47.510257 4697 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:47.510416 4697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:47.510843 4697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:47.514156 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "ffca23bafcd948269e4f40ba6a065910"
format_stamp: "Formatted at 2026-08-06 09:04:47 on dist-test-slave-8tdl"
server_key: "300e05257a52d4a8accef04d485ce9a1"
server_key_iv: "98383fabf581d990c299cc69fa33fec3"
server_key_version: "encryptionkey@0"
I20260806 09:04:47.515200 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "ffca23bafcd948269e4f40ba6a065910"
format_stamp: "Formatted at 2026-08-06 09:04:47 on dist-test-slave-8tdl"
server_key: "300e05257a52d4a8accef04d485ce9a1"
server_key_iv: "98383fabf581d990c299cc69fa33fec3"
server_key_version: "encryptionkey@0"
I20260806 09:04:47.521436 4697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:04:47.526362 4713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:47.528218 4697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:04:47.528458 4697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ffca23bafcd948269e4f40ba6a065910"
format_stamp: "Formatted at 2026-08-06 09:04:47 on dist-test-slave-8tdl"
server_key: "300e05257a52d4a8accef04d485ce9a1"
server_key_iv: "98383fabf581d990c299cc69fa33fec3"
server_key_version: "encryptionkey@0"
I20260806 09:04:47.528712 4697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:47.555214 4697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:47.560385 4697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:47.560703 4697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:47.561847 4697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:47.563534 4697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:47.563656 4697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:47.563812 4697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:47.563884 4697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:47.603134 4697 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:45999
I20260806 09:04:47.603196 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:45999 every 8 connection(s)
I20260806 09:04:47.604573 4697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:47.614653 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4697
I20260806 09:04:47.614917 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:04:47.615409 28398 external_mini_cluster.cc:1428] Setting key 1a242f0f5078fe8286e4da676276c38b
Aug 06 09:04:47 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007087, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:47.629487 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:47.608093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:44917 (local address 127.27.187.190:45697)
0806 09:04:47.608691 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:47.608697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:47.609660 (+ 963us) server_negotiation.cc:407] Connection header received
0806 09:04:47.611659 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:47.611663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:47.611737 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:47.611840 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:47.613468 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.614437 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.616401 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.616936 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.617044 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:47.621354 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:47.621365 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:47.621372 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:47.621425 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:47.625469 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:47.626475 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:47.626484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:47.626490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:47.626713 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:47.627202 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:47.627210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:47.627214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:47.627627 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:47.627791 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:47.628709 (+ 918us) server_negotiation.cc:299] Negotiation successful
0806 09:04:47.628992 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
I20260806 09:04:47.630631 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:47.608730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:44917)
0806 09:04:47.609479 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:47.609519 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:47.611362 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:47.612076 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:47.612085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:47.612677 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:47.613265 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:47.613282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.614623 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.614627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:47.616250 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:47.616264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:47.617293 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:47.617302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:47.617540 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:47.618288 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:47.618313 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:47.621034 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:47.625696 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:47.625705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:47.625723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:47.626302 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:47.626852 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:47.626855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:47.626858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:47.627068 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:47.627782 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:47.627789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:47.628219 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 09:04:47.628556 (+ 337us) client_negotiation.cc:240] Negotiation successful
0806 09:04:47.629021 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":295,"threads_started":1}
I20260806 09:04:47.632521 4826 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45697
I20260806 09:04:47.633015 4826 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:47.633886 4826 heartbeater.cc:511] Master 127.27.187.190:45697 requested a full tablet report, sending...
I20260806 09:04:47.635737 4498 ts_manager.cc:194] Registered new tserver with Master: ffca23bafcd948269e4f40ba6a065910 (127.27.187.130:45999)
I20260806 09:04:47.637121 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:44917
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:04:47.683493 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:48.032790 4691 heartbeater.cc:503] Master 127.27.187.190:45697 was elected leader, sending a full tablet report...
W20260806 09:04:48.062294 4833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:48.062618 4833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:48.062849 4833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:48.070482 4833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:48.070616 4833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:48.070812 4833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:48.081316 4833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:48.082666 4833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:48.084164 4833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:48.098822 4839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:48.099002 4838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:48.099200 4841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:48.100553 4833 server_base.cc:1034] running on GCE node
I20260806 09:04:48.101377 4833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:48.102473 4833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:48.103768 4833 hybrid_clock.cc:648] HybridClock initialized: now 1786007088103706 us; error 49 us; skew 500 ppm
I20260806 09:04:48.104188 4833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:48 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007088, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:48.108090 4833 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:04:48.109913 4833 webserver.cc:460] Webserver started at http://127.27.187.131:35749/ using document root <none> and password file <none>
I20260806 09:04:48.110762 4833 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:48.110919 4833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:48.111274 4833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:04:48.114511 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "eb846df5e39a4a84aefa114861197c9f"
format_stamp: "Formatted at 2026-08-06 09:04:48 on dist-test-slave-8tdl"
server_key: "acca4b4a86acb81c314f4be550a6244e"
server_key_iv: "62368c5eb2a2fe001697ab534cfefd4d"
server_key_version: "encryptionkey@0"
I20260806 09:04:48.115541 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "eb846df5e39a4a84aefa114861197c9f"
format_stamp: "Formatted at 2026-08-06 09:04:48 on dist-test-slave-8tdl"
server_key: "acca4b4a86acb81c314f4be550a6244e"
server_key_iv: "62368c5eb2a2fe001697ab534cfefd4d"
server_key_version: "encryptionkey@0"
I20260806 09:04:48.122062 4833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:04:48.126416 4848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:48.128130 4833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:04:48.128350 4833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "eb846df5e39a4a84aefa114861197c9f"
format_stamp: "Formatted at 2026-08-06 09:04:48 on dist-test-slave-8tdl"
server_key: "acca4b4a86acb81c314f4be550a6244e"
server_key_iv: "62368c5eb2a2fe001697ab534cfefd4d"
server_key_version: "encryptionkey@0"
I20260806 09:04:48.128587 4833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:48.152185 4833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:48.157102 4833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:48.157519 4833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:48.158969 4833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:48.160502 4833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:48.160619 4833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:48.160774 4833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:48.160846 4833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:48.200254 4833 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33105
I20260806 09:04:48.200305 4961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33105 every 8 connection(s)
I20260806 09:04:48.201815 4833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:48.207347 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4833
I20260806 09:04:48.207535 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:04:48.208035 28398 external_mini_cluster.cc:1428] Setting key 86e06160ac8692361b6561cf7a8c0e64
Aug 06 09:04:48 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007088, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:04:48.225899 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:48.204926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:33077 (local address 127.27.187.190:45697)
0806 09:04:48.205486 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:48.205491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:48.206474 (+ 983us) server_negotiation.cc:407] Connection header received
0806 09:04:48.208710 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:48.208715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:48.208832 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:48.208942 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:48.210903 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.211932 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.213870 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.214285 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.214397 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:48.218839 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:48.218845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:48.218848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:48.218900 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:48.221875 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.223079 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.223085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.223092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.223279 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.223902 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.223906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.223908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.224206 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:48.224386 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:48.225239 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 09:04:48.225475 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":355,"threads_started":1}
I20260806 09:04:48.226996 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:48.205457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:33077)
0806 09:04:48.206275 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:48.206319 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:48.208356 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:48.209259 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:48.209269 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:48.209921 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:48.210666 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:48.210749 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.212167 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.212173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:48.213685 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:48.213701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.214524 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.214530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:48.214841 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:48.215570 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:48.215598 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:48.218334 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:04:48.222061 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:48.222073 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:48.222095 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:48.222912 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:48.223426 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:04:48.223433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:04:48.223439 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:04:48.223751 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:04:48.224359 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:04:48.224368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:04:48.224850 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 09:04:48.225143 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 09:04:48.225723 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":328,"threads_started":1}
I20260806 09:04:48.228624 4962 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45697
I20260806 09:04:48.228972 4962 heartbeater.cc:464] Registering TS with master...
I20260806 09:04:48.229915 4962 heartbeater.cc:511] Master 127.27.187.190:45697 requested a full tablet report, sending...
I20260806 09:04:48.231735 4498 ts_manager.cc:194] Registered new tserver with Master: eb846df5e39a4a84aefa114861197c9f (127.27.187.131:33105)
I20260806 09:04:48.232807 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:33077
I20260806 09:04:48.237041 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
Entry for principal oryx/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab.
WARNING: no policy specified for oryx/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
Entry for principal oryx/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab.
WARNING: no policy specified for oryx/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
Entry for principal oryx/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab.
I20260806 09:04:48.430814 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4560
Aug 06 09:04:48 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:04:48.450102 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:48.432350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.27.187.129:39861)
0806 09:04:48.433129 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:48.433137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:48.433186 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:04:48.433288 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:48.433296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:48.433616 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:48.433798 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:48.435535 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.436666 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.438052 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.438790 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.438905 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:48.443634 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:48.443651 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:48.443669 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:48.443742 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:48.447218 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.448008 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.448015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.448018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.448170 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.448598 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.448602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.448604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.448982 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:48.449236 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:48.449514 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 09:04:48.449807 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":351,"threads_started":1}
I20260806 09:04:48.451478 4686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:48.539490 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4560
I20260806 09:04:48.559149 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4697
Aug 06 09:04:48 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:04:48.578285 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:48.560190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.27.187.130:45999)
0806 09:04:48.560768 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:48.560775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:48.560802 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:04:48.560861 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:48.560867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:48.561084 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:48.561219 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:48.562617 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.563803 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.565348 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.566061 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.566188 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:48.571062 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:48.571077 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:48.571094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:48.571144 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:48.574494 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.575646 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.575655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.575658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.575830 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.576581 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.576585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.576587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.576986 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:48.577336 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:48.577667 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 09:04:48.577893 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260806 09:04:48.579926 4821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:48.667968 4826 heartbeater.cc:503] Master 127.27.187.190:45697 was elected leader, sending a full tablet report...
I20260806 09:04:48.668967 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4697
W20260806 09:04:48.684952 4483 connection.cc:535] server connection from 127.27.187.130:44917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:04:48.685129 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4833
Aug 06 09:04:48 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:04:48.704099 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:48.686241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.27.187.131:33105)
0806 09:04:48.686910 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:48.686917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:48.686948 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:04:48.687007 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:48.687013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:48.687296 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:48.687436 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:48.688784 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.689779 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.691216 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:48.692070 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:48.692212 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:48.697206 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:48.697222 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:48.697239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:48.697283 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:48.700827 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.701702 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.701710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.701712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.701849 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:48.702295 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:48.702299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:48.702302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:48.702750 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:48.703069 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:48.703397 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 09:04:48.703698 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":246,"threads_started":1}
I20260806 09:04:48.705518 4957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:48.792394 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4833
I20260806 09:04:48.810590 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4466
I20260806 09:04:48.812124 4529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:04:48.922896 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4466
I20260806 09:04:48.947314 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45697
--webserver_interface=127.27.187.190
--webserver_port=38225
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:04:49.326130 4981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:49.326529 4981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:49.326745 4981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:49.334455 4981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:04:49.334664 4981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:49.334811 4981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:49.334923 4981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:04:49.335016 4981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:04:49.350016 4981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=38225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:49.352139 4981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:49.354287 4981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:49.372690 4986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:49.372601 4989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:49.372634 4987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:49.373692 4981 server_base.cc:1034] running on GCE node
I20260806 09:04:49.374616 4981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:49.376389 4981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:49.377673 4981 hybrid_clock.cc:648] HybridClock initialized: now 1786007089377603 us; error 74 us; skew 500 ppm
I20260806 09:04:49.378064 4981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:49 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007089, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:49.387266 4981 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:04:49.389127 4981 webserver.cc:460] Webserver started at http://127.27.187.190:38225/ using document root <none> and password file <none>
I20260806 09:04:49.389916 4981 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:49.390096 4981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:49.396337 4981 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:49.399828 4996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:49.401722 4981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:49.401968 4981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "1ae65e32d926439ab62188d62a51b2a6"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "41c8167f28b6e990f55e5881ac8b5e74"
server_key_iv: "91b01b3aa2af9b517e63e59a81bb35ab"
server_key_version: "encryptionkey@0"
I20260806 09:04:49.403298 4981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:49.441597 4981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:49.447180 4981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:49.447492 4981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:49.469559 4981 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45697
I20260806 09:04:49.469604 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45697 every 8 connection(s)
I20260806 09:04:49.471479 4981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:04:49.474040 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 4981
I20260806 09:04:49.475138 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:39861
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:45697
--webserver_port=35773
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:04:49.482218 5048 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:04:49.487764 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Bootstrap starting.
I20260806 09:04:49.508736 5048 log.cc:824] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:04:49.518414 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:04:49.519045 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Bootstrap complete.
I20260806 09:04:49.529022 5048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:49.529948 5048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae65e32d926439ab62188d62a51b2a6, State: Initialized, Role: FOLLOWER
I20260806 09:04:49.530642 5048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:49.530961 5048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:04:49.531100 5048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:04:49.531299 5048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:04:49.535521 5048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:49.536114 5048 leader_election.cc:302] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae65e32d926439ab62188d62a51b2a6; no voters:
I20260806 09:04:49.536818 5048 leader_election.cc:288] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:04:49.536970 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:04:49.538359 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [term 2 LEADER]: Becoming Leader. State: Replica: 1ae65e32d926439ab62188d62a51b2a6, State: Running, Role: LEADER
I20260806 09:04:49.539079 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } }
I20260806 09:04:49.539494 5048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:04:49.542093 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae65e32d926439ab62188d62a51b2a6. Latest consensus state: current_term: 2 leader_uuid: "1ae65e32d926439ab62188d62a51b2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } } }
I20260806 09:04:49.542543 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:49.542505 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ae65e32d926439ab62188d62a51b2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae65e32d926439ab62188d62a51b2a6" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45697 } } }
I20260806 09:04:49.542851 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:04:49.553725 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:04:49.560446 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:04:49.562047 5058 catalog_manager.cc:1259] Loaded cluster ID: fb85bc0c815f415dbed6b8ff66a4e2f3
I20260806 09:04:49.562247 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:04:49.566567 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:04:49.568252 5058 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ae65e32d926439ab62188d62a51b2a6: Loaded TSK: 0
I20260806 09:04:49.570185 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:04:49.863512 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:49.863840 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:49.863992 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:49.872061 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:49.872195 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:49.872380 5050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:04:49.883333 5050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.129.keytab
--principal=oryx/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:39861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=35773
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:49.884783 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:49.886344 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:49.901355 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:49.901402 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:49.902098 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:49.902644 5050 server_base.cc:1034] running on GCE node
I20260806 09:04:49.903384 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:49.904527 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:49.905872 5050 hybrid_clock.cc:648] HybridClock initialized: now 1786007089905762 us; error 46 us; skew 500 ppm
I20260806 09:04:49.906227 5050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:49 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007089, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:49.910313 5050 init.cc:377] Logged in from keytab as oryx/127.27.187.129@KRBTEST.COM (short username oryx)
I20260806 09:04:49.912269 5050 webserver.cc:460] Webserver started at http://127.27.187.129:35773/ using document root <none> and password file <none>
I20260806 09:04:49.913079 5050 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:49.913265 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:49.920334 5050 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:04:49.923893 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:49.925767 5050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:04:49.926270 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "67b96efea5174a2ab2e7270e377856c3"
format_stamp: "Formatted at 2026-08-06 09:04:46 on dist-test-slave-8tdl"
server_key: "e99484bd5b512001ea6ac360880a577a"
server_key_iv: "f53be893ae6136d0e24c269b556a77b8"
server_key_version: "encryptionkey@0"
I20260806 09:04:49.927523 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:49.953368 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:49.958354 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:49.958666 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:49.959867 5050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:49.961444 5050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:49.961581 5050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:49.961735 5050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:49.961826 5050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:50.000921 5050 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:39861
I20260806 09:04:50.000986 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:39861 every 8 connection(s)
I20260806 09:04:50.002722 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:04:50.008524 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5050
I20260806 09:04:50.009364 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:45999
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:45697
--webserver_port=41117
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:50.026546 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.006313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:59303)
0806 09:04:50.007285 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.007345 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.010154 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.012027 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.012058 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.012971 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.013894 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.013927 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.015906 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.015915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.018228 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.018260 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.019713 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.019724 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.020068 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.020981 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.021026 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.025495 (+ 4469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":421,"threads_started":1}
W20260806 09:04:50.027442 5201 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.006323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:59303 (local address 127.27.187.190:45697)
0806 09:04:50.007637 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.007650 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.007699 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:04:50.010935 (+ 3236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.010950 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.011385 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.011767 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.014251 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.015695 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.018475 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.019395 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.019571 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.026023 (+ 6452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:59303: BlockingRecv error: recv got EOF from 127.27.187.129:59303 (error 108)
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":836,"threads_started":1}
W20260806 09:04:50.027562 5198 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:45697 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.187.190:45697: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:50.038720 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.028253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:37425)
0806 09:04:50.028458 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.028484 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.028582 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.028971 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.028974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.029374 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.029525 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.029533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.031180 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.031185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.032416 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.032433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.033586 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.033591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.033776 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.034617 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.034639 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.038297 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 09:04:50.039448 5201 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.028351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:37425 (local address 127.27.187.190:45697)
0806 09:04:50.028562 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.028571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.028597 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:04:50.028702 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.028710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.028811 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.028941 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.029684 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.030969 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.032590 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.033402 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.033554 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.039044 (+ 5490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:37425: BlockingRecv error: recv got EOF from 127.27.187.129:37425 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:50.048537 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.039569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:48029)
0806 09:04:50.040010 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.040031 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.040158 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.040552 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.040558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.040988 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.041147 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.041156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.042233 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.042239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.043271 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.043283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.044162 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.044169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.044366 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.045124 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.045143 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.048188 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292}
W20260806 09:04:50.048954 5198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:04:50.049114 5201 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.039844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:48029 (local address 127.27.187.190:45697)
0806 09:04:50.040031 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.040038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.040087 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:04:50.040323 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.040328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.040414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.040523 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.041304 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.042070 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.043429 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.043962 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.044110 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.048506 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:48029: BlockingRecv error: recv got EOF from 127.27.187.129:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 09:04:50.382565 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:50.382961 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:50.383062 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:50.390564 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:50.390765 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:50.390909 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:04:50.401631 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.130.keytab
--principal=oryx/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:45999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=41117
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:50.403687 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:50.405146 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:50.427156 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:50.427263 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:50.427225 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:50.431205 5202 server_base.cc:1034] running on GCE node
I20260806 09:04:50.432086 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:50.433336 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:50.434590 5202 hybrid_clock.cc:648] HybridClock initialized: now 1786007090434523 us; error 62 us; skew 500 ppm
I20260806 09:04:50.435029 5202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007090, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:50.439086 5202 init.cc:377] Logged in from keytab as oryx/127.27.187.130@KRBTEST.COM (short username oryx)
I20260806 09:04:50.441179 5202 webserver.cc:460] Webserver started at http://127.27.187.130:41117/ using document root <none> and password file <none>
I20260806 09:04:50.442020 5202 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:50.442199 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:50.449101 5202 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:04:50.452754 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:50.454656 5202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:50.454941 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ffca23bafcd948269e4f40ba6a065910"
format_stamp: "Formatted at 2026-08-06 09:04:47 on dist-test-slave-8tdl"
server_key: "300e05257a52d4a8accef04d485ce9a1"
server_key_iv: "98383fabf581d990c299cc69fa33fec3"
server_key_version: "encryptionkey@0"
I20260806 09:04:50.456208 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:50.492780 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:50.498476 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:50.498847 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:50.499962 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:50.501552 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:50.501698 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:50.501832 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:50.501920 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:50.547843 5202 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:45999
I20260806 09:04:50.547886 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:45999 every 8 connection(s)
I20260806 09:04:50.549608 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:04:50.556417 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5202
I20260806 09:04:50.557269 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:33105
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:45697
--webserver_port=35749
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:04:50.576834 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.553064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:51535 (local address 127.27.187.190:45697)
0806 09:04:50.559128 (+ 6064us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.559138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.559164 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:04:50.559241 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.559248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.559354 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.559562 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.562579 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.563814 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.567462 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.568267 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.568426 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.576432 (+ 8006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:51535: BlockingRecv error: recv got EOF from 127.27.187.130:51535 (error 108)
Metrics: {"server-negotiator.queue_time_us":5888,"thread_start_us":5733,"threads_started":1}
I20260806 09:04:50.578239 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.553726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:51535)
0806 09:04:50.554769 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.554879 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.557809 (+ 2930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.559960 (+ 2151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.559978 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.561105 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.562194 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.562232 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.564121 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.564129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.567209 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.567235 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.568580 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.568590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.568983 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.570089 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.570137 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.576104 (+ 5967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":379,"threads_started":1}
W20260806 09:04:50.579190 5332 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:45697 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.187.190:45697: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:04:50.593312 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.580109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:58873 (local address 127.27.187.190:45697)
0806 09:04:50.580335 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.580343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.580478 (+ 135us) server_negotiation.cc:407] Connection header received
0806 09:04:50.580774 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.580783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.580877 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.580994 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.582055 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.583477 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.585746 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.586629 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.586829 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.593036 (+ 6207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:58873: BlockingRecv error: recv got EOF from 127.27.187.130:58873 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:04:50.594058 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.579972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:58873)
0806 09:04:50.580356 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.580381 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.580623 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.581046 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.581052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.581618 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.581862 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.581879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.583664 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.583672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.585529 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.585562 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.586878 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.586887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.587109 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.587910 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.587940 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.593737 (+ 5797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218}
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:50.607050 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:50.595128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:37481)
0806 09:04:50.595505 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:50.595534 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:50.595654 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:50.596134 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:50.596140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:50.596596 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:50.596829 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.596847 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.598726 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.598732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.600333 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:50.600351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.601664 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.601672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:50.601902 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.602787 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:50.602813 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:50.606744 (+ 3931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252}
W20260806 09:04:50.607530 5332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:04:50.607684 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:50.595410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:37481 (local address 127.27.187.190:45697)
0806 09:04:50.595628 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:50.595636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:50.595664 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:04:50.595857 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:50.595865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:50.595966 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:50.596086 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:50.597027 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.598504 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.600523 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:50.601415 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:50.601631 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:50.607121 (+ 5490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:37481: BlockingRecv error: recv got EOF from 127.27.187.130:37481 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 09:04:50.941143 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:04:50.941527 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:04:50.941704 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:04:50.949397 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:04:50.949595 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:04:50.949800 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:04:50.960655 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/krb5kdc/oryx_127.27.187.131.keytab
--principal=oryx/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:33105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=35749
--tserver_master_addrs=127.27.187.190:45697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:04:50.962229 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:04:50.963728 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:04:50.979796 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:50.979794 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:04:50.980032 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:04:50.981580 5336 server_base.cc:1034] running on GCE node
I20260806 09:04:50.982546 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:04:50.983820 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:04:50.985176 5336 hybrid_clock.cc:648] HybridClock initialized: now 1786007090985121 us; error 55 us; skew 500 ppm
I20260806 09:04:50.985562 5336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:04:50 dist-test-slave-8tdl krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007090, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:04:50.989465 5336 init.cc:377] Logged in from keytab as oryx/127.27.187.131@KRBTEST.COM (short username oryx)
I20260806 09:04:50.991385 5336 webserver.cc:460] Webserver started at http://127.27.187.131:35749/ using document root <none> and password file <none>
I20260806 09:04:50.992319 5336 fs_manager.cc:359] Metadata directory not provided
I20260806 09:04:50.992496 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:04:50.998919 5336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 09:04:51.002892 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:51.004972 5336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:04:51.005187 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "eb846df5e39a4a84aefa114861197c9f"
format_stamp: "Formatted at 2026-08-06 09:04:48 on dist-test-slave-8tdl"
server_key: "acca4b4a86acb81c314f4be550a6244e"
server_key_iv: "62368c5eb2a2fe001697ab534cfefd4d"
server_key_version: "encryptionkey@0"
I20260806 09:04:51.006604 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:04:51.022831 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:04:51.027671 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:04:51.027925 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:04:51.028955 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:04:51.030503 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:04:51.030601 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:04:51.030761 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:04:51.030828 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:51.062790 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.049876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:58571)
0806 09:04:51.050496 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:51.050524 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:51.050752 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:51.051179 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:51.051185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:51.051930 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:51.052209 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.052239 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.054072 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.054079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.056124 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.056145 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.057322 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.057334 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.057537 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.058668 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:51.058750 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:51.062466 (+ 3716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
W20260806 09:04:51.063269 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:51.050021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:58571 (local address 127.27.187.190:45697)
0806 09:04:51.050273 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.050279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.050636 (+ 357us) server_negotiation.cc:407] Connection header received
0806 09:04:51.050925 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.050931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.051019 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.051141 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.052444 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.053797 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.056329 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.056998 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.057159 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.062891 (+ 5732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:58571: BlockingRecv error: recv got EOF from 127.27.187.129:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:04:51.071817 5336 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33105
I20260806 09:04:51.071883 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33105 every 8 connection(s)
I20260806 09:04:51.073508 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:04:51.074481 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5336
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:04:51.095726 5469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:51.076574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:40561 (local address 127.27.187.190:45697)
0806 09:04:51.077607 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.077618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.078517 (+ 899us) server_negotiation.cc:407] Connection header received
0806 09:04:51.081366 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.081373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.081479 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.081603 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.083830 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.084801 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.087230 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.087949 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.088118 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.095298 (+ 7180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:40561: BlockingRecv error: recv got EOF from 127.27.187.131:40561 (error 108)
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":795,"threads_started":1}
I20260806 09:04:51.095778 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.075952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.27.187.190:45697)
0806 09:04:51.076146 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.076150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.076768 (+ 618us) server_negotiation.cc:407] Connection header received
0806 09:04:51.077000 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.077006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.077094 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.077228 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.078953 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.080346 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.082288 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.082944 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.083087 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.086642 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:04:51.086651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:04:51.086697 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:04:51.086792 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:04:51.090596 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:51.091730 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:51.091748 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:51.091759 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:51.092102 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:04:51.093004 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:04:51.093008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:04:51.093011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:04:51.093778 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:04:51.094061 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 09:04:51.094821 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 09:04:51.095350 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:04:51.096355 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.077343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:40561)
0806 09:04:51.078332 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:51.078382 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:51.081030 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:51.081890 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:51.081899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:51.082809 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:51.083612 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.083640 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.085000 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.085009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.087066 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.087080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.088259 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.088265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.088490 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.089287 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:51.089313 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:51.094933 (+ 5620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":413,"threads_started":1}
W20260806 09:04:51.097224 5466 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:45697 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.187.190:45697: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:51.108180 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.097903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:57933)
0806 09:04:51.098094 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:51.098111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:51.098207 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:51.098599 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:51.098607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:51.099089 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:51.099257 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.099269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.100984 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.100989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.101987 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.102003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.103047 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.103053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.103254 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.103983 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:51.104003 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:51.107820 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 09:04:51.109005 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:51.097965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:57933 (local address 127.27.187.190:45697)
0806 09:04:51.098223 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.098229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.098252 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:04:51.098308 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.098313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.098399 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.098509 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.099771 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.100821 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.102303 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.102876 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.102983 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.108101 (+ 5118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:57933: BlockingRecv error: recv got EOF from 127.27.187.131:57933 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:51.119768 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.109239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:58463)
0806 09:04:51.109482 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:51.109502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:51.109647 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:51.110085 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:51.110092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:51.110491 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:51.110745 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.110758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.112183 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.112188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.113629 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.113855 (+ 226us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.114852 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.114858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.115043 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.115780 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:51.115801 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:51.119507 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260806 09:04:51.120172 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:51.109563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:58463 (local address 127.27.187.190:45697)
0806 09:04:51.109777 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.109782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.109807 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:04:51.109859 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.109863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.109933 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.110041 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.110960 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.111971 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.114019 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.114585 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.114811 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.119848 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:58463: BlockingRecv error: recv got EOF from 127.27.187.131:58463 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 09:04:51.120252 5466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:51 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:51.618885 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:51.608381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:38467)
0806 09:04:51.609127 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:51.609155 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:51.609322 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:51.609739 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:51.609744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:51.610295 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:51.610545 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.610556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.612006 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.612010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.613164 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:51.613179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.614054 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.614060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:51.614235 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.614968 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:51.614989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:51.618463 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":460,"threads_started":1}
W20260806 09:04:51.619138 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:51.608434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:38467 (local address 127.27.187.190:45697)
0806 09:04:51.608645 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:51.608651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:51.609292 (+ 641us) server_negotiation.cc:407] Connection header received
0806 09:04:51.609425 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:51.609429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:51.609503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:51.609601 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:51.610781 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.611834 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.613358 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:51.613826 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:51.613945 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:51.618881 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:38467: BlockingRecv error: recv got EOF from 127.27.187.130:38467 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:52.073742 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:52.064176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:39433)
0806 09:04:52.064836 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:52.064856 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:52.065066 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:52.065523 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:52.065528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:52.066137 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:52.066382 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.066394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.067649 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.067655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.068631 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.068645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.069415 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.069421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.069611 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.070301 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:52.070321 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:52.073382 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":363,"threads_started":1}
W20260806 09:04:52.073875 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:52.064231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:39433 (local address 127.27.187.190:45697)
0806 09:04:52.064493 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:52.064498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:52.064979 (+ 481us) server_negotiation.cc:407] Connection header received
0806 09:04:52.065269 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:52.065273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:52.065345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:52.065492 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:52.066578 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.067472 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.068793 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.069241 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.069395 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.073636 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:39433: BlockingRecv error: recv got EOF from 127.27.187.129:39433 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:52.130491 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:52.120959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:57881)
0806 09:04:52.121531 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:52.121551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:52.121728 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:52.122092 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:52.122097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:52.122615 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:52.122841 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.122850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.123903 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.123909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.124989 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.125004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.125839 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.125846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.126032 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.126716 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:52.126736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:52.130174 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
W20260806 09:04:52.130825 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:52.121048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:57881 (local address 127.27.187.190:45697)
0806 09:04:52.121350 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:52.121359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:52.121657 (+ 298us) server_negotiation.cc:407] Connection header received
0806 09:04:52.121866 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:52.121875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:52.121938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:52.122044 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:52.123008 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.123752 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.125159 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.125640 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.125787 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.130533 (+ 4746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:57881: BlockingRecv error: recv got EOF from 127.27.187.131:57881 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:52 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:52.630589 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:52.620137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:42573)
0806 09:04:52.620819 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:52.620836 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:52.621022 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:52.621482 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:52.621487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:52.621956 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:52.622132 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.622142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.623348 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.623355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.624426 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:52.624439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.625249 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.625255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:52.625427 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.626448 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:52.626502 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:52.630335 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":337,"threads_started":1}
W20260806 09:04:52.630959 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:52.620464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:42573 (local address 127.27.187.190:45697)
0806 09:04:52.620725 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:52.620733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:52.620934 (+ 201us) server_negotiation.cc:407] Connection header received
0806 09:04:52.621238 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:52.621242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:52.621329 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:52.621445 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:52.622299 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.623209 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.624576 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:52.625087 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:52.625226 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:52.630607 (+ 5381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:42573: BlockingRecv error: recv got EOF from 127.27.187.130:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:53.086036 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:53.075153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:48287)
0806 09:04:53.075937 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:53.075959 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:53.076233 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:53.076654 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:53.076659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:53.077083 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:53.077316 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.077328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.078649 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.078654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.080134 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.080151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.081134 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.081140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.081420 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.082094 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:53.082114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:53.085671 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":386,"threads_started":1}
W20260806 09:04:53.086234 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:53.075432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:48287 (local address 127.27.187.190:45697)
0806 09:04:53.075732 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:53.075740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:53.076134 (+ 394us) server_negotiation.cc:407] Connection header received
0806 09:04:53.076427 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:53.076431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:53.076501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:53.076620 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:53.077491 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.078431 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.080339 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.080971 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.081124 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.085976 (+ 4852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:48287: BlockingRecv error: recv got EOF from 127.27.187.129:48287 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:53.141721 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:53.131760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:39749)
0806 09:04:53.132410 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:53.132428 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:53.132623 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:53.133058 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:53.133063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:53.133655 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:53.133842 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.133854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.135088 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.135094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.136030 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.136044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.136840 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.136848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.137026 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.137726 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:53.137746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:53.141412 (+ 3666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":394,"threads_started":1}
W20260806 09:04:53.142200 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:53.131933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:39749 (local address 127.27.187.190:45697)
0806 09:04:53.132195 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:53.132201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:53.132532 (+ 331us) server_negotiation.cc:407] Connection header received
0806 09:04:53.132768 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:53.132776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:53.132873 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:53.133017 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:53.134018 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.134928 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.136196 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.136663 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.136812 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.141739 (+ 4927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:39749: BlockingRecv error: recv got EOF from 127.27.187.131:39749 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:53 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:53.641647 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:53.631998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:52359)
0806 09:04:53.632688 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:53.632708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:53.632835 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:53.633410 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:53.633417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:53.633826 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:53.634036 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.634046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.635462 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.635469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.636518 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:53.636533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.637302 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.637308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:53.637491 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.638149 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:53.638168 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:53.641358 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":339,"threads_started":1}
W20260806 09:04:53.641911 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:53.632773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:52359 (local address 127.27.187.190:45697)
0806 09:04:53.632976 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:53.632981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:53.632999 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:04:53.633059 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:53.633062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:53.633147 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:53.633239 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:53.634227 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.635322 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.636694 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:53.637149 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:53.637288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:53.641653 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:52359: BlockingRecv error: recv got EOF from 127.27.187.130:52359 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:54.096535 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:54.087470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:34923)
0806 09:04:54.088005 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:54.088022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:54.088193 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:54.088603 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:54.088618 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:54.089021 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:54.089178 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.089188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.090231 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.090238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.091314 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.091327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.092064 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.092070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.092256 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.092940 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:54.092959 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:54.096293 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":299,"threads_started":1}
W20260806 09:04:54.096820 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:54.087798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34923 (local address 127.27.187.190:45697)
0806 09:04:54.087967 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:54.087972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:54.088148 (+ 176us) server_negotiation.cc:407] Connection header received
0806 09:04:54.088370 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:54.088374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:54.088441 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:54.088554 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:54.089335 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.090077 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.091469 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.091891 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.092032 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.096518 (+ 4486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:34923: BlockingRecv error: recv got EOF from 127.27.187.129:34923 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:54.152479 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:54.143028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:44657)
0806 09:04:54.143689 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:54.143707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:54.143831 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:54.144201 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:54.144204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:54.144645 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:54.144826 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.144835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.146092 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.146097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.147121 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.147136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.147899 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.147904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.148035 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.148741 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:54.148760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:54.151986 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":382,"threads_started":1}
W20260806 09:04:54.152650 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:54.143091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:44657 (local address 127.27.187.190:45697)
0806 09:04:54.143354 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:54.143359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:54.143800 (+ 441us) server_negotiation.cc:407] Connection header received
0806 09:04:54.143988 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:54.143993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:54.144061 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:54.144155 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:54.144997 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.145917 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.147286 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.147739 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.147858 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.152307 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:44657: BlockingRecv error: recv got EOF from 127.27.187.131:44657 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:54 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:54.657779 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:54.642828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:39595)
0806 09:04:54.643428 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:54.643453 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:54.643627 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:54.644009 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:54.644015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:54.644509 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:54.644681 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.644691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.645789 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.645795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.646816 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:54.646829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.647614 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.647619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:54.647802 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.653909 (+ 6107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:54.653934 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:54.657527 (+ 3593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
W20260806 09:04:54.658103 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:54.642880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:39595 (local address 127.27.187.190:45697)
0806 09:04:54.643101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:54.643109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:54.643559 (+ 450us) server_negotiation.cc:407] Connection header received
0806 09:04:54.643764 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:54.643770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:54.643870 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:54.643975 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:54.644840 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.645612 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.646973 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:54.647445 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:54.647585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:54.657785 (+ 10200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:39595: BlockingRecv error: recv got EOF from 127.27.187.130:39595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:55.107538 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:55.097897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:46261)
0806 09:04:55.098651 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:55.098669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:55.098887 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:55.099367 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:55.099371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:55.099900 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:55.100076 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.100089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.101226 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.101230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.102240 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.102254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.103061 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.103065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.103217 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.103883 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:55.103902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:55.107122 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":405,"threads_started":1}
W20260806 09:04:55.107676 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:55.097955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:46261 (local address 127.27.187.190:45697)
0806 09:04:55.098205 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:55.098214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:55.098855 (+ 641us) server_negotiation.cc:407] Connection header received
0806 09:04:55.099079 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:55.099087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:55.099206 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:55.099303 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:55.100250 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.101063 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.102406 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.102917 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.103020 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.107388 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:46261: BlockingRecv error: recv got EOF from 127.27.187.129:46261 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:55.163707 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:55.153697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:52291)
0806 09:04:55.154242 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:55.154259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:55.154380 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:55.154855 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:55.154859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:55.155382 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:55.155555 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.155567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.156726 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.156730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.157972 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.157988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.159125 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.159131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.159313 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.160158 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:55.160178 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:55.163451 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
W20260806 09:04:55.163990 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:55.153876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:52291 (local address 127.27.187.190:45697)
0806 09:04:55.154082 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:55.154087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:55.154352 (+ 265us) server_negotiation.cc:407] Connection header received
0806 09:04:55.154487 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:55.154491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:55.154627 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:55.154751 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:55.155750 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.156557 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.158193 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.158955 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.159091 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.163715 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:52291: BlockingRecv error: recv got EOF from 127.27.187.131:52291 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:55 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:55.668594 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:55.658935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:36275)
0806 09:04:55.659539 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:55.659556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:55.659673 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:55.660014 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:55.660019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:55.660722 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:55.660891 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.660901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.661992 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.661996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.663042 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:55.663056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.663919 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.663923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:55.664294 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.664975 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:55.664996 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:55.668335 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":340,"threads_started":1}
W20260806 09:04:55.668989 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:55.659238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:36275 (local address 127.27.187.190:45697)
0806 09:04:55.659434 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:55.659439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:55.659654 (+ 215us) server_negotiation.cc:407] Connection header received
0806 09:04:55.659759 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:55.659762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:55.659839 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:55.659923 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:55.661049 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.661830 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.663308 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:55.663756 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:55.663882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:55.668731 (+ 4849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:36275: BlockingRecv error: recv got EOF from 127.27.187.130:36275 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:56.118789 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:56.108932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:53569)
0806 09:04:56.109911 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:56.109933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:56.110098 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:56.110511 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:56.110516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:56.111136 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:56.111304 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.111315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.112532 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.112537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.113477 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.113490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.114272 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.114277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.114466 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.115186 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:56.115204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:56.118357 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":684,"threads_started":1}
W20260806 09:04:56.118944 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:56.109358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:53569 (local address 127.27.187.190:45697)
0806 09:04:56.109588 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:56.109596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:56.110048 (+ 452us) server_negotiation.cc:407] Connection header received
0806 09:04:56.110249 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:56.110255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:56.110342 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:56.110476 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:56.111575 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.112342 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.113644 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.114077 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.114237 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.118618 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:53569: BlockingRecv error: recv got EOF from 127.27.187.129:53569 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:56.174494 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:56.164842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:36627)
0806 09:04:56.165423 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:56.165441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:56.165563 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:56.165938 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:56.165943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:56.166473 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:56.166706 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.166719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.167821 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.167825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.168884 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.168897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.170144 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.170148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.170284 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.171066 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:56.171086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:56.174185 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
W20260806 09:04:56.174899 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:56.164950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:36627 (local address 127.27.187.190:45697)
0806 09:04:56.165229 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:56.165236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:56.165541 (+ 305us) server_negotiation.cc:407] Connection header received
0806 09:04:56.165679 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:56.165683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:56.165760 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:56.165854 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:56.166859 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.167660 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.169064 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.169938 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.170043 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.174582 (+ 4539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:36627: BlockingRecv error: recv got EOF from 127.27.187.131:36627 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:56 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:56.680617 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:56.669887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:34411)
0806 09:04:56.670513 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:56.670530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:56.670653 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:56.671133 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:56.671139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:56.671684 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:56.671869 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.671884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.673188 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.673196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.674232 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:56.674248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.675478 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.675483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:56.675625 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.676455 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:56.676485 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:56.680300 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
W20260806 09:04:56.680866 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:56.669998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:34411 (local address 127.27.187.190:45697)
0806 09:04:56.670333 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:56.670338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:56.670628 (+ 290us) server_negotiation.cc:407] Connection header received
0806 09:04:56.670846 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:56.670851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:56.670939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:56.671037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:56.672070 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.672954 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.674434 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:56.675240 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:56.675444 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:56.680569 (+ 5125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:34411: BlockingRecv error: recv got EOF from 127.27.187.130:34411 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:57.129662 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:57.120066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:60961)
0806 09:04:57.120787 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:57.120807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:57.120973 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:57.121430 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:57.121436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:57.121880 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:57.122277 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.122302 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.123495 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.123499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.124542 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.124558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.125297 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.125302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.125430 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.126240 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:57.126259 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:57.129378 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":317,"threads_started":1}
W20260806 09:04:57.129958 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:57.120278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:60961 (local address 127.27.187.190:45697)
0806 09:04:57.120540 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:57.120547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:57.120910 (+ 363us) server_negotiation.cc:407] Connection header received
0806 09:04:57.121135 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:57.121141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:57.121230 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:57.121325 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:57.122460 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.123337 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.124698 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.125113 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.125222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.129636 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:60961: BlockingRecv error: recv got EOF from 127.27.187.129:60961 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:57.185796 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:57.175754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:58597)
0806 09:04:57.176316 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:57.176333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:57.176461 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:57.176924 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:57.176928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:57.177413 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:57.177586 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.177597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.178936 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.178940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.179907 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.179921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.180815 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.180820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.180963 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.181781 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:57.181801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:57.185473 (+ 3672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
W20260806 09:04:57.186115 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:57.175831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:58597 (local address 127.27.187.190:45697)
0806 09:04:57.176100 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:57.176108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:57.176435 (+ 327us) server_negotiation.cc:407] Connection header received
0806 09:04:57.176615 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:57.176653 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:57.176724 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:57.176815 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:57.177776 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.178705 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.180170 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.180624 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.180738 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.185791 (+ 5053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:58597: BlockingRecv error: recv got EOF from 127.27.187.131:58597 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:57 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:57.691103 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:57.681767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:46987)
0806 09:04:57.682461 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:57.682478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:57.682713 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:57.683125 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:57.683131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:57.683606 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:57.683795 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.683805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.684827 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.684833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.685792 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:57.685804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.686510 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.686516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:57.686766 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.687404 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:57.687423 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:57.690836 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":302,"threads_started":1}
W20260806 09:04:57.691469 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:57.681887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:46987 (local address 127.27.187.190:45697)
0806 09:04:57.682305 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:57.682312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:57.682574 (+ 262us) server_negotiation.cc:407] Connection header received
0806 09:04:57.682891 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:57.682895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:57.682972 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:57.683085 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:57.683951 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.684679 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.685939 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:57.686350 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:57.686485 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:57.691073 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:46987: BlockingRecv error: recv got EOF from 127.27.187.130:46987 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:58.140091 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:58.130840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:36483)
0806 09:04:58.131532 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:58.131554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:58.131693 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:58.131996 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:58.131999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:58.132545 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:58.132699 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.132710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.133805 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.133809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.134819 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.134831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.135577 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.135581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.135698 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.136350 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:58.136369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:58.139846 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":313,"threads_started":1}
W20260806 09:04:58.140527 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:58.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:36483 (local address 127.27.187.190:45697)
0806 09:04:58.131070 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:58.131078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:58.131645 (+ 567us) server_negotiation.cc:407] Connection header received
0806 09:04:58.131804 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:58.131808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:58.131872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:58.131974 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:58.132831 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.133662 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.134949 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.135409 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.135507 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.140168 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:36483: BlockingRecv error: recv got EOF from 127.27.187.129:36483 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:58.197997 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:58.187101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:56485)
0806 09:04:58.187803 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:58.187829 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:58.187999 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:58.188426 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:58.188429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:58.188877 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:58.189062 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.189071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.190282 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.190287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.191505 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.191521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.192426 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.192433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.192784 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.193746 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:58.193776 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:58.197722 (+ 3946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":291,"threads_started":1}
W20260806 09:04:58.198287 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:58.187330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:56485 (local address 127.27.187.190:45697)
0806 09:04:58.187614 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:58.187622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:58.187955 (+ 333us) server_negotiation.cc:407] Connection header received
0806 09:04:58.188158 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:58.188163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:58.188280 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:58.188366 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:58.189257 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.190099 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.191715 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.192226 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.192393 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.197986 (+ 5593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:56485: BlockingRecv error: recv got EOF from 127.27.187.131:56485 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:58 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:58.702226 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:58.692321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:44125)
0806 09:04:58.693005 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:58.693022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:58.693159 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:58.693578 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:58.693591 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:58.694160 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:58.694324 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.694335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.695570 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.695574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.696599 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:58.696612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.697387 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.697392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:58.697570 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.698470 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:58.698490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:58.701978 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":409,"threads_started":1}
W20260806 09:04:58.702512 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:58.692425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:44125 (local address 127.27.187.190:45697)
0806 09:04:58.692732 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:58.692742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:58.693113 (+ 371us) server_negotiation.cc:407] Connection header received
0806 09:04:58.693275 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:58.693279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:58.693361 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:58.693445 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:58.694473 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.695416 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.696742 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:58.697172 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:58.697273 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:58.702201 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:44125: BlockingRecv error: recv got EOF from 127.27.187.130:44125 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:59.151921 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:59.141415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:48577)
0806 09:04:59.142108 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:59.142127 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:59.142333 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:59.142810 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:59.142815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:59.143320 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:59.143600 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.143615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.145007 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.145014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.146193 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.146208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.147130 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.147135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.147337 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.148154 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:59.148177 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:59.151527 (+ 3350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":412,"threads_started":1}
W20260806 09:04:59.152029 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:59.141540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:48577 (local address 127.27.187.190:45697)
0806 09:04:59.141943 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:59.141951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:59.142247 (+ 296us) server_negotiation.cc:407] Connection header received
0806 09:04:59.142492 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:59.142503 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:59.142579 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:59.142727 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:59.143818 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.144842 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.146371 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.146940 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.147079 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.151787 (+ 4708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:48577: BlockingRecv error: recv got EOF from 127.27.187.129:48577 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:59.208756 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:59.199071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:50933)
0806 09:04:59.199740 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:59.199757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:59.199879 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:59.200325 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:59.200329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:59.200760 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:59.200930 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.200939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.202110 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.202114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.203118 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.203131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.203924 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.203929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.204044 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.204817 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:59.204836 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:59.208352 (+ 3516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":351,"threads_started":1}
W20260806 09:04:59.208806 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:59.199361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:50933 (local address 127.27.187.190:45697)
0806 09:04:59.199548 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:59.199552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:59.199869 (+ 317us) server_negotiation.cc:407] Connection header received
0806 09:04:59.200022 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:59.200026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:59.200092 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:59.200266 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:59.201089 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.201949 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.203290 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.203712 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.203824 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.208555 (+ 4731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:50933: BlockingRecv error: recv got EOF from 127.27.187.131:50933 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:04:59 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:04:59.718918 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:04:59.703287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:33981)
0806 09:04:59.703842 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 09:04:59.703858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:04:59.703980 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:04:59.704480 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:04:59.704485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:04:59.704911 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:04:59.705083 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.705092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.706270 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.706274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.707455 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:04:59.707469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.708578 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.708584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:04:59.708730 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.714528 (+ 5798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:04:59.714552 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:04:59.718509 (+ 3957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":324,"threads_started":1}
W20260806 09:04:59.719094 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:04:59.703393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:33981 (local address 127.27.187.190:45697)
0806 09:04:59.703573 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:04:59.703577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:04:59.704012 (+ 435us) server_negotiation.cc:407] Connection header received
0806 09:04:59.704085 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:04:59.704089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:04:59.704228 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:04:59.704329 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:04:59.705331 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.706084 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.707695 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:04:59.708355 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:04:59.708541 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:04:59.718781 (+ 10240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:33981: BlockingRecv error: recv got EOF from 127.27.187.130:33981 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:00.163769 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:00.153072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:44527)
0806 09:05:00.153819 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:00.153844 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:00.154011 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:00.154410 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:00.154415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:00.155192 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:00.155376 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.155387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.156600 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.156605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.157678 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.157695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.158829 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.158836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.158986 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.159758 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:00.159777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:00.163487 (+ 3710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":447,"threads_started":1}
W20260806 09:05:00.164528 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:00.153264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:44527 (local address 127.27.187.190:45697)
0806 09:05:00.153520 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:00.153530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:00.153941 (+ 411us) server_negotiation.cc:407] Connection header received
0806 09:05:00.154125 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:00.154128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:00.154218 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:00.154303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:00.155561 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.156424 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.157883 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.158532 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.158647 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.163814 (+ 5167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:44527: BlockingRecv error: recv got EOF from 127.27.187.129:44527 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:00.219901 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:00.209939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:56209)
0806 09:05:00.210596 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:00.210612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:00.210818 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:00.211196 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:00.211201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:00.211788 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:00.211971 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.211981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.213043 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.213049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.214092 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.214106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.214947 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.214953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.215143 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.215842 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:00.215881 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:00.219625 (+ 3744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":387,"threads_started":1}
W20260806 09:05:00.220232 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:00.210004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:56209 (local address 127.27.187.190:45697)
0806 09:05:00.210307 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:00.210316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:00.210742 (+ 426us) server_negotiation.cc:407] Connection header received
0806 09:05:00.210964 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:00.210968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:00.211042 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:00.211150 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:00.212150 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.212895 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.214264 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.214781 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.214919 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.219979 (+ 5060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:56209: BlockingRecv error: recv got EOF from 127.27.187.131:56209 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:00 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:00.730865 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:00.720269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:40975)
0806 09:05:00.721280 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:00.721300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:00.721432 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:00.721891 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:00.721899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:00.722564 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:00.722802 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.722820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.724014 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.724020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.725590 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:00.725615 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.726458 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.726468 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:00.726660 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.727446 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:00.727470 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:00.730529 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":930,"threads_started":1}
W20260806 09:05:00.731207 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:00.720382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:40975 (local address 127.27.187.190:45697)
0806 09:05:00.720664 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:00.720671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:00.721382 (+ 711us) server_negotiation.cc:407] Connection header received
0806 09:05:00.721585 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:00.721593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:00.721733 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:00.721853 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:00.722994 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.723854 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.725773 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:00.726283 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:00.726392 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:00.730916 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:40975: BlockingRecv error: recv got EOF from 127.27.187.130:40975 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:01.175340 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:01.165270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:34421)
0806 09:05:01.165954 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:01.165974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:01.166162 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:01.166592 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:01.166597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:01.167148 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:01.167343 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.167354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.168599 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.168605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.169629 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.169644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.170495 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.170501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.170784 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.171525 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:01.171554 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:01.174922 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":390,"threads_started":1}
W20260806 09:05:01.175465 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:01.165536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34421 (local address 127.27.187.190:45697)
0806 09:05:01.165789 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:01.165797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:01.166064 (+ 267us) server_negotiation.cc:407] Connection header received
0806 09:05:01.166354 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:01.166359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:01.166428 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:01.166545 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:01.167525 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.168439 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.169807 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.170313 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.170465 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.175196 (+ 4731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:34421: BlockingRecv error: recv got EOF from 127.27.187.129:34421 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:01.231696 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:01.220897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:37131)
0806 09:05:01.221465 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:01.221481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:01.221595 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:01.221981 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:01.221985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:01.222390 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:01.222576 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.222587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.223856 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.223861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.225051 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.225069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.226179 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.226194 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.226340 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.227211 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:01.227231 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:01.231036 (+ 3805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":343,"threads_started":1}
W20260806 09:05:01.232447 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:01.221033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:37131 (local address 127.27.187.190:45697)
0806 09:05:01.221275 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:01.221280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:01.221580 (+ 300us) server_negotiation.cc:407] Connection header received
0806 09:05:01.221725 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:01.221730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:01.221798 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:01.221886 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:01.222805 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.223666 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.225314 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.225770 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.225883 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.231578 (+ 5695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:37131: BlockingRecv error: recv got EOF from 127.27.187.131:37131 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:01 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:01.742054 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:01.731955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:54241)
0806 09:05:01.732548 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:01.732570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:01.732712 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:01.733301 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:01.733306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:01.733760 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:01.733967 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.733978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.735305 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.735308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.736422 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:01.736436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.737520 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.737526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:01.737720 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.738431 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:01.738451 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:01.741781 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":455,"threads_started":1}
W20260806 09:05:01.742273 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:01.732373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:54241 (local address 127.27.187.190:45697)
0806 09:05:01.732618 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:01.732626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:01.732653 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:05:01.732854 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:01.732861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:01.732980 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:01.733144 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:01.734229 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.735148 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.736577 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:01.737338 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:01.737494 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:01.742014 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:54241: BlockingRecv error: recv got EOF from 127.27.187.130:54241 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:02.186127 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:02.176545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:46573)
0806 09:05:02.177386 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:02.177416 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:02.177602 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:02.177990 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:02.177994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:02.178429 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:02.178589 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.178599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.179685 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.179691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.180638 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.180660 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.181420 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.181426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.181621 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.182454 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:02.182475 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:02.185705 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":648,"threads_started":1}
W20260806 09:05:02.186265 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:02.176764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:46573 (local address 127.27.187.190:45697)
0806 09:05:02.177024 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:02.177036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:02.177525 (+ 489us) server_negotiation.cc:407] Connection header received
0806 09:05:02.177750 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:02.177757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:02.177833 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:02.177947 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:02.178789 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.179517 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.180802 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.181260 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.181400 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.185988 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:46573: BlockingRecv error: recv got EOF from 127.27.187.129:46573 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:02.242511 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:02.232893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:38735)
0806 09:05:02.233480 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:02.233497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:02.233626 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:02.233940 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:02.233944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:02.234420 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:02.234614 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.234625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.235891 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.235895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.236851 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.236865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.237678 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.237683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.237835 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.238590 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:02.238609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:02.242146 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":345,"threads_started":1}
W20260806 09:05:02.242609 5334 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:02.233021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:38735 (local address 127.27.187.190:45697)
0806 09:05:02.233327 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:02.233333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:02.233638 (+ 305us) server_negotiation.cc:407] Connection header received
0806 09:05:02.233723 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:02.233726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:02.233791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:02.233872 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:02.234844 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.235723 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.237007 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.237462 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.237628 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.242371 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:38735: BlockingRecv error: recv got EOF from 127.27.187.131:38735 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:02 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:02.756399 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:02.743493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:59721)
0806 09:05:02.744053 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:02.744077 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:02.744241 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:02.745405 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:02.745412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:02.745995 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:02.746231 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.746253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.748660 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.748667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.750205 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:02.750225 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.751347 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.751356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:02.751579 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.752365 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:02.752386 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:02.756115 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":403,"threads_started":1}
W20260806 09:05:02.756850 5506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:02.744080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:59721 (local address 127.27.187.190:45697)
0806 09:05:02.744894 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:02.744906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:02.744924 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:05:02.744980 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:02.744984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:02.745055 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:02.745195 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:02.746433 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.748280 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.750400 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:02.751124 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:02.751269 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:02.756428 (+ 5159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:59721: BlockingRecv error: recv got EOF from 127.27.187.130:59721 (error 108)
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":328,"threads_started":1}
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:03.197543 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:03.187512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:42605)
0806 09:05:03.188127 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:03.188144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:03.188290 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:03.188687 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:03.188691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:03.189091 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:03.189333 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.189344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.190880 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.190885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.192279 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.192298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.193049 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.193054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.193174 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.193911 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:03.193931 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:03.197150 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":331,"threads_started":1}
W20260806 09:05:03.197657 5506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:03.187629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:42605 (local address 127.27.187.190:45697)
0806 09:05:03.187851 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:03.187856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:03.188272 (+ 416us) server_negotiation.cc:407] Connection header received
0806 09:05:03.188434 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:03.188438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:03.188527 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:03.188640 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:03.189514 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.190656 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.192447 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.192888 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.192989 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.197397 (+ 4408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:42605: BlockingRecv error: recv got EOF from 127.27.187.129:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:03.252940 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:03.243700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:50655)
0806 09:05:03.244286 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:03.244303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:03.244435 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:03.244803 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:03.244807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:03.245278 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:03.245440 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.245448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.246593 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.246597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.247568 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.247580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.248479 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.248484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.248628 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.249408 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:03.249429 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:03.252689 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":320,"threads_started":1}
W20260806 09:05:03.253314 5506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:03.243723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:50655 (local address 127.27.187.190:45697)
0806 09:05:03.244004 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:03.244015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:03.244417 (+ 402us) server_negotiation.cc:407] Connection header received
0806 09:05:03.244553 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:03.244557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:03.244643 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:03.244728 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:03.245610 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.246437 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.247749 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.248297 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.248406 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.252962 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:50655: BlockingRecv error: recv got EOF from 127.27.187.131:50655 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:03 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:03.767561 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:03.757620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:52267)
0806 09:05:03.758361 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:03.758387 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:03.758542 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:03.759031 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:03.759058 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:03.759525 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:03.759697 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.759707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.760887 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.760895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.761955 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:03.761970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.762828 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.762834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:03.763031 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.763698 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:03.763717 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:03.767299 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":374,"threads_started":1}
W20260806 09:05:03.768138 5510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:03.757815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:52267 (local address 127.27.187.190:45697)
0806 09:05:03.758618 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:03.758624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:03.758643 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:05:03.758733 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:03.758738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:03.758863 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:03.758989 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:03.759870 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.760721 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.762130 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:03.762588 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:03.762760 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:03.767634 (+ 4874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:52267: BlockingRecv error: recv got EOF from 127.27.187.130:52267 (error 108)
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":554,"threads_started":1}
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:04.209195 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:04.198880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:51105)
0806 09:05:04.199589 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:04.199608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:04.199743 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:04.200322 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:04.200330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:04.200744 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:04.200948 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.200959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.202152 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.202158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.203342 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.203356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.204232 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.204238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.204407 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.205280 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:04.205303 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:04.208771 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":475,"threads_started":1}
W20260806 09:05:04.209401 5510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:04.198959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:51105 (local address 127.27.187.190:45697)
0806 09:05:04.199173 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:04.199180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:04.199712 (+ 532us) server_negotiation.cc:407] Connection header received
0806 09:05:04.199909 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:04.199916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:04.200009 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:04.200136 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:04.201118 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.201963 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.203514 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.204004 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.204173 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.209075 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:51105: BlockingRecv error: recv got EOF from 127.27.187.129:51105 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:04.264451 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:04.254054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:46029)
0806 09:05:04.254713 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:04.254734 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:04.254977 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:04.255587 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:04.255594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:04.256197 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:04.256453 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.256464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.257842 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.257848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.258881 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.258894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.259650 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.259657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.259876 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.260609 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:04.260635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:04.264025 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":291,"threads_started":1}
W20260806 09:05:04.265131 5510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:04.254093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:46029 (local address 127.27.187.190:45697)
0806 09:05:04.254522 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:04.254529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:04.254837 (+ 308us) server_negotiation.cc:407] Connection header received
0806 09:05:04.255146 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:04.255154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:04.255268 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:04.255378 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:04.256604 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.257659 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.259031 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.259477 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.259618 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.264779 (+ 5161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:46029: BlockingRecv error: recv got EOF from 127.27.187.131:46029 (error 108)
Metrics: {"server-negotiator.queue_time_us":315}
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:04 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:04.783238 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:04.768646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:52375)
0806 09:05:04.769348 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:04.769377 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:04.769519 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:04.769968 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:04.769973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:04.770458 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:04.770653 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.770666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.771984 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.771989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.772961 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:04.772974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.773869 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.773874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:04.774023 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.779164 (+ 5141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:04.779193 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:04.782954 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":394,"threads_started":1}
W20260806 09:05:04.783802 5514 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:04.768830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:52375 (local address 127.27.187.190:45697)
0806 09:05:04.769333 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:04.769338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:04.769462 (+ 124us) server_negotiation.cc:407] Connection header received
0806 09:05:04.769704 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:04.769708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:04.769778 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:04.769878 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:04.770888 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.771804 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.773207 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:04.773677 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:04.773809 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:04.783517 (+ 9708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:52375: BlockingRecv error: recv got EOF from 127.27.187.130:52375 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.129@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:05.220709 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:05.210843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.129:50907)
0806 09:05:05.211566 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:05.211584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:05.211724 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:05.212254 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:05.212258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:05.212679 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:05.212849 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.212857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.214024 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.214028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.215116 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.215130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.215912 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.215916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.216033 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.216716 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:05.216734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:05.220429 (+ 3695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":452,"threads_started":1}
W20260806 09:05:05.220948 5514 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:05.211049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:50907 (local address 127.27.187.190:45697)
0806 09:05:05.211313 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:05.211318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:05.211679 (+ 361us) server_negotiation.cc:407] Connection header received
0806 09:05:05.211870 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:05.211875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:05.212002 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:05.212224 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:05.213006 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.213860 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.215304 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.215756 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.215870 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.220712 (+ 4842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.129:50907: BlockingRecv error: recv got EOF from 127.27.187.129:50907 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.131@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:05.276258 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:05.265852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.131:50111)
0806 09:05:05.266391 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:05.266408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:05.266532 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:05.266987 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:05.266991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:05.267468 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:05.267697 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.267708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.268854 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.268858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.269899 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.269913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.270977 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.270982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.271145 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.272187 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:05.272216 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:05.275993 (+ 3777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
W20260806 09:05:05.276675 5514 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:05.265986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:50111 (local address 127.27.187.190:45697)
0806 09:05:05.266224 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:05.266229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:05.266528 (+ 299us) server_negotiation.cc:407] Connection header received
0806 09:05:05.266693 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:05.266698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:05.266782 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:05.266886 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:05.267869 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.268675 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.270071 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.270796 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.270947 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.276367 (+ 5420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.131:50111: BlockingRecv error: recv got EOF from 127.27.187.131:50111 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:05:05 dist-test-slave-8tdl krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.187.130@KRBTEST.COM for oryx/127.27.187.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:05:05.795056 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:05.784833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45697 (local address 127.27.187.130:40199)
0806 09:05:05.785403 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:05.785421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:05.785534 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:05.786066 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:05.786071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:05.786513 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:05.786854 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.786872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.788197 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.788202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.789422 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:05.789437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.790360 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.790366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:05.790542 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.791473 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:05.791494 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:05.794773 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.187.190:45697: Server oryx/127.27.187.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":284,"threads_started":1}
W20260806 09:05:05.795346 5518 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:05.784982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:40199 (local address 127.27.187.190:45697)
0806 09:05:05.785758 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:05.785763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:05.785781 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:05:05.785834 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:05.785837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:05.785897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:05.785998 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:05.787039 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.788017 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.789581 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:05.790192 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:05.790343 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:05.795020 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.187.130:40199: BlockingRecv error: recv got EOF from 127.27.187.130:40199 (error 108)
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":523,"threads_started":1}
I20260806 09:05:06.075680 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5050
I20260806 09:05:06.090497 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:06.077132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.27.187.129:39861)
0806 09:05:06.077824 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:06.077832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:06.077863 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:05:06.077931 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:06.077939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:06.078190 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:06.078492 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:06.080402 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.081522 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.083206 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.083838 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.083963 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:06.086972 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:06.086982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:06.087000 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:06.087048 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:06.089937 (+ 2889us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:05:06.090262 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60372: Request ticket server kudu/127.27.187.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":365,"threads_started":1}
W20260806 09:05:06.090607 5520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60372: Request ticket server kudu/127.27.187.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:05:06.091017 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5050: Not authorized: Client connection negotiation failed: client connection to 127.27.187.129:39861: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:05:06.091303 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5050
I20260806 09:05:06.112126 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5202
I20260806 09:05:06.127372 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:06.113546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.27.187.130:45999)
0806 09:05:06.114289 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:06.114297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:06.114330 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:05:06.114396 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:06.114403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:06.114641 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:06.114927 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:06.116388 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.117551 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.119069 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.119734 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.119900 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:06.122822 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:06.122834 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:06.122850 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:06.122912 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:06.126309 (+ 3397us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:05:06.126941 (+ 632us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33998: Request ticket server kudu/127.27.187.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":341,"threads_started":1}
W20260806 09:05:06.127512 5521 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33998: Request ticket server kudu/127.27.187.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:05:06.127471 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5202: Not authorized: Client connection negotiation failed: client connection to 127.27.187.130:45999: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:05:06.127674 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5202
I20260806 09:05:06.148761 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5336
I20260806 09:05:06.164391 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:06.150001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.27.187.131:33105)
0806 09:05:06.150765 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:06.150773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:06.150800 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:05:06.150865 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:06.150871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:06.151148 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:06.151318 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:06.152801 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.154029 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.155652 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.156290 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.156396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:06.160484 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:06.160495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:06.160511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:06.160555 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:06.163774 (+ 3219us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:05:06.164118 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33040: Request ticket server kudu/127.27.187.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":410,"threads_started":1}
W20260806 09:05:06.164525 5522 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33040: Request ticket server kudu/127.27.187.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:05:06.164585 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5336: Not authorized: Client connection negotiation failed: client connection to 127.27.187.131:33105: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.187.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:05:06.164825 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5336
I20260806 09:05:06.185169 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 4981
I20260806 09:05:06.186878 5043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:06.311615 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 4981
2026-08-06T09:05:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21983 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T09:05:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:05:06Z Disabled control of system clock
I20260806 09:05:06.410969 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:36265
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:36265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:05:06.771644 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:06.771977 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:06.772079 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:06.779956 5527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:05:06.780110 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:06.780193 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:06.780299 5527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:05:06.780378 5527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:05:06.790243 5527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:36265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:36265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:06.791880 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:06.793813 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:06.807931 5535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:06.808027 5532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:06.808053 5533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:06.808419 5527 server_base.cc:1034] running on GCE node
I20260806 09:05:06.809618 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:06.811398 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:06.812708 5527 hybrid_clock.cc:648] HybridClock initialized: now 1786007106812653 us; error 43 us; skew 500 ppm
I20260806 09:05:06.813220 5527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:06.816111 5527 webserver.cc:460] Webserver started at http://127.27.187.190:37835/ using document root <none> and password file <none>
I20260806 09:05:06.816936 5527 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:06.817088 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:06.817468 5527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:06.820708 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "23cda12be7464e5b8a3bbba19be8e332"
format_stamp: "Formatted at 2026-08-06 09:05:06 on dist-test-slave-8tdl"
server_key: "a11a6a6a0a6614112e2aae1495378c0c"
server_key_iv: "1941b4c5114ecf70b86e2160f0a297a9"
server_key_version: "encryptionkey@0"
I20260806 09:05:06.821588 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "23cda12be7464e5b8a3bbba19be8e332"
format_stamp: "Formatted at 2026-08-06 09:05:06 on dist-test-slave-8tdl"
server_key: "a11a6a6a0a6614112e2aae1495378c0c"
server_key_iv: "1941b4c5114ecf70b86e2160f0a297a9"
server_key_version: "encryptionkey@0"
I20260806 09:05:06.827350 5527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 09:05:06.831552 5541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:06.833365 5527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:05:06.833607 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "23cda12be7464e5b8a3bbba19be8e332"
format_stamp: "Formatted at 2026-08-06 09:05:06 on dist-test-slave-8tdl"
server_key: "a11a6a6a0a6614112e2aae1495378c0c"
server_key_iv: "1941b4c5114ecf70b86e2160f0a297a9"
server_key_version: "encryptionkey@0"
I20260806 09:05:06.833884 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:06.860678 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:06.861799 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:06.862174 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:06.884055 5527 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:36265
I20260806 09:05:06.884066 5592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:36265 every 8 connection(s)
I20260806 09:05:06.885936 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:05:06.890209 5593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:06.893064 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5527
I20260806 09:05:06.893276 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:05:06.894111 28398 external_mini_cluster.cc:1428] Setting key 8b304040204c3e3b0400843ebf1da626
I20260806 09:05:06.903450 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332: Bootstrap starting.
I20260806 09:05:06.906935 5593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:06.908497 5593 log.cc:824] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:06.909682 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:06.896815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.27.187.190:36265)
0806 09:05:06.897924 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:06.897936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:06.897981 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:05:06.899807 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:06.899832 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:06.900387 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:06.900866 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:06.901527 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.902919 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.904382 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:06.905107 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:06.905326 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:06.906609 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:06.906617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:06.906633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:06.907348 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:06.907469 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:06.907493 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:06.907698 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:06.907931 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 09:05:06.908309 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":346,"threads_started":1}
I20260806 09:05:06.912295 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332: No bootstrap required, opened a new log
I20260806 09:05:06.920473 5593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } }
I20260806 09:05:06.920943 5593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:06.921063 5593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23cda12be7464e5b8a3bbba19be8e332, State: Initialized, Role: FOLLOWER
I20260806 09:05:06.921679 5593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } }
I20260806 09:05:06.921936 5593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:05:06.922091 5593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:05:06.922266 5593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:06.924856 5593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } }
I20260806 09:05:06.925436 5593 leader_election.cc:302] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23cda12be7464e5b8a3bbba19be8e332; no voters:
I20260806 09:05:06.926126 5593 leader_election.cc:288] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:05:06.926328 5598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:05:06.927345 5598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [term 1 LEADER]: Becoming Leader. State: Replica: 23cda12be7464e5b8a3bbba19be8e332, State: Running, Role: LEADER
I20260806 09:05:06.927971 5598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } }
I20260806 09:05:06.928732 5593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:05:06.931340 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23cda12be7464e5b8a3bbba19be8e332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } } }
I20260806 09:05:06.931500 5600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23cda12be7464e5b8a3bbba19be8e332. Latest consensus state: current_term: 1 leader_uuid: "23cda12be7464e5b8a3bbba19be8e332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23cda12be7464e5b8a3bbba19be8e332" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 36265 } } }
I20260806 09:05:06.931995 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:06.931993 5600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:06.934487 5607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:05:06.939836 5607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:05:06.949885 5607 catalog_manager.cc:1347] Generated new cluster ID: b2612279cdf44d2da4b03be128263104
I20260806 09:05:06.950031 5607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:05:06.965368 5607 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:05:06.967410 5607 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:05:06.978004 5607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23cda12be7464e5b8a3bbba19be8e332: Generated new TSK 0
I20260806 09:05:06.979501 5607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:05:06.996551 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--builtin_ntp_servers=127.27.187.148:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 09:05:07.375355 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:07.375684 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:07.375844 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:07.384542 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:07.384680 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:07.384877 5617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:05:07.395268 5617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:07.396720 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:07.398891 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:07.414877 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:07.414808 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:07.415087 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:07.415742 5617 server_base.cc:1034] running on GCE node
I20260806 09:05:07.416616 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:07.417861 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:07.419175 5617 hybrid_clock.cc:648] HybridClock initialized: now 1786007107419046 us; error 60 us; skew 500 ppm
I20260806 09:05:07.419641 5617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:07.422526 5617 webserver.cc:460] Webserver started at http://127.27.187.129:41017/ using document root <none> and password file <none>
I20260806 09:05:07.423431 5617 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:07.423533 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:07.423882 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:07.427271 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "e5ee403bccb34af7992bc4cee57f251e"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "5d11aeda01ff57a13896d1ecfaac9fb2"
server_key_iv: "249cc35c3be8ce842da5a8764031191b"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.428282 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ee403bccb34af7992bc4cee57f251e"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "5d11aeda01ff57a13896d1ecfaac9fb2"
server_key_iv: "249cc35c3be8ce842da5a8764031191b"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.434530 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:05:07.439049 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:07.440815 5617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:05:07.441025 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "e5ee403bccb34af7992bc4cee57f251e"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "5d11aeda01ff57a13896d1ecfaac9fb2"
server_key_iv: "249cc35c3be8ce842da5a8764031191b"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.441287 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:07.470629 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:07.471776 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:07.472147 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:07.473344 5617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:07.475015 5617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:07.475111 5617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:07.475337 5617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:07.475442 5617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:07.512989 5617 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:39645
I20260806 09:05:07.513125 5743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:39645 every 8 connection(s)
I20260806 09:05:07.514431 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:05:07.522430 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5617
I20260806 09:05:07.522716 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:05:07.523425 28398 external_mini_cluster.cc:1428] Setting key 773b84f02bd57d8b12bcfbc6d086b598
I20260806 09:05:07.526201 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--builtin_ntp_servers=127.27.187.148:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:05:07.528651 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:07.517356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:40739 (local address 127.27.187.190:36265)
0806 09:05:07.517876 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:07.517881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:07.519227 (+ 1346us) server_negotiation.cc:407] Connection header received
0806 09:05:07.521147 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:07.521153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:07.521256 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:07.521380 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:07.522521 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:07.523416 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:07.525078 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:07.525503 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:07.525684 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:07.527429 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:07.527439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:07.527446 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:07.527469 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:07.527545 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:07.527562 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:07.527677 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:07.528040 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 09:05:07.528184 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260806 09:05:07.529862 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:07.517965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36265 (local address 127.27.187.129:40739)
0806 09:05:07.519050 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:07.519089 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:07.520838 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:07.521626 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:07.521635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:07.521733 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:07.522328 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:07.522346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:07.523598 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:07.523603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:07.524901 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:07.524915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:07.525781 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:07.525789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:07.526043 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:07.526888 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:07.526891 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:07.526936 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:07.527083 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:07.527702 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:07.527733 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:07.527742 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:05:07.527993 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 09:05:07.528468 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":366,"threads_started":1}
I20260806 09:05:07.531718 5744 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36265
I20260806 09:05:07.532061 5744 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:07.532867 5744 heartbeater.cc:511] Master 127.27.187.190:36265 requested a full tablet report, sending...
I20260806 09:05:07.535657 5558 ts_manager.cc:194] Registered new tserver with Master: e5ee403bccb34af7992bc4cee57f251e (127.27.187.129:39645)
I20260806 09:05:07.537955 5558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.129:40739
W20260806 09:05:07.888695 5748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:07.889094 5748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:07.889225 5748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:07.901304 5748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:07.901465 5748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:07.901638 5748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:05:07.913328 5748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:07.914762 5748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:07.916885 5748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:07.932837 5753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:07.932971 5754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:07.932919 5756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:07.934482 5748 server_base.cc:1034] running on GCE node
I20260806 09:05:07.935262 5748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:07.936458 5748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:07.937829 5748 hybrid_clock.cc:648] HybridClock initialized: now 1786007107937705 us; error 67 us; skew 500 ppm
I20260806 09:05:07.938373 5748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:07.941413 5748 webserver.cc:460] Webserver started at http://127.27.187.130:46079/ using document root <none> and password file <none>
I20260806 09:05:07.942274 5748 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:07.942422 5748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:07.942896 5748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:07.946326 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "0c76acbc72ac4e478ac1990dfacabd7d"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "a72d4e30d0df66652b1eaa7f1943ae9c"
server_key_iv: "17b409c20fc17b9d54b510f33a3826f9"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.947376 5748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "0c76acbc72ac4e478ac1990dfacabd7d"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "a72d4e30d0df66652b1eaa7f1943ae9c"
server_key_iv: "17b409c20fc17b9d54b510f33a3826f9"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.953612 5748 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:05:07.957772 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:07.959573 5748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:05:07.959802 5748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "0c76acbc72ac4e478ac1990dfacabd7d"
format_stamp: "Formatted at 2026-08-06 09:05:07 on dist-test-slave-8tdl"
server_key: "a72d4e30d0df66652b1eaa7f1943ae9c"
server_key_iv: "17b409c20fc17b9d54b510f33a3826f9"
server_key_version: "encryptionkey@0"
I20260806 09:05:07.960063 5748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:07.987607 5748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:07.988586 5748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:07.988941 5748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:07.990044 5748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:07.991727 5748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:07.991865 5748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:07.992007 5748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:07.992101 5748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:08.032146 5748 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:35089
I20260806 09:05:08.032269 5874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:35089 every 8 connection(s)
I20260806 09:05:08.033737 5748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:05:08.040628 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5748
I20260806 09:05:08.040926 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:05:08.041429 28398 external_mini_cluster.cc:1428] Setting key 8d07641afaf54c4f01348055336984b6
I20260806 09:05:08.044602 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--builtin_ntp_servers=127.27.187.148:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:05:08.048277 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.036654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:57487 (local address 127.27.187.190:36265)
0806 09:05:08.037150 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.037155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.038524 (+ 1369us) server_negotiation.cc:407] Connection header received
0806 09:05:08.040505 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.040511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.040611 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.040737 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.041964 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.042949 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.044799 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.045353 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.045526 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.047219 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:08.047224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:08.047228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:08.047248 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:08.047291 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:08.047305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:08.047410 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:08.047693 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 09:05:08.047804 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
I20260806 09:05:08.049333 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.037207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36265 (local address 127.27.187.130:57487)
0806 09:05:08.038346 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:08.038395 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:08.040218 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:08.040977 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:08.040985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:08.041128 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:08.041782 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:08.041801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.043120 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.043124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:08.044635 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:08.044649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.045621 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.045628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:08.045885 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.046842 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:08.046844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:08.046889 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:08.046987 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:08.047410 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:08.047424 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:08.047435 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:05:08.047682 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 09:05:08.048156 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":326,"threads_started":1}
I20260806 09:05:08.051124 5875 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36265
I20260806 09:05:08.051504 5875 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:08.052583 5875 heartbeater.cc:511] Master 127.27.187.190:36265 requested a full tablet report, sending...
I20260806 09:05:08.054467 5558 ts_manager.cc:194] Registered new tserver with Master: 0c76acbc72ac4e478ac1990dfacabd7d (127.27.187.130:35089)
I20260806 09:05:08.055919 5558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.130:57487
W20260806 09:05:08.409363 5879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:08.409682 5879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:08.409776 5879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:08.418071 5879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:08.418273 5879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:08.418390 5879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:05:08.428232 5879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:36265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:08.429803 5879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:08.431797 5879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:08.446501 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:08.446421 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:08.446375 5887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:08.447918 5879 server_base.cc:1034] running on GCE node
I20260806 09:05:08.448568 5879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:08.449579 5879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:08.450835 5879 hybrid_clock.cc:648] HybridClock initialized: now 1786007108450789 us; error 34 us; skew 500 ppm
I20260806 09:05:08.451188 5879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:08.453869 5879 webserver.cc:460] Webserver started at http://127.27.187.131:45925/ using document root <none> and password file <none>
I20260806 09:05:08.454640 5879 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:08.454813 5879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:08.455240 5879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:08.458436 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "396ab2b6a705468dbd0c21b8d16a361b"
format_stamp: "Formatted at 2026-08-06 09:05:08 on dist-test-slave-8tdl"
server_key: "83a7022094eebc52b504e1d9a71437db"
server_key_iv: "db5d0d45864f3f24afcecfb49e60bd1c"
server_key_version: "encryptionkey@0"
I20260806 09:05:08.459336 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "396ab2b6a705468dbd0c21b8d16a361b"
format_stamp: "Formatted at 2026-08-06 09:05:08 on dist-test-slave-8tdl"
server_key: "83a7022094eebc52b504e1d9a71437db"
server_key_iv: "db5d0d45864f3f24afcecfb49e60bd1c"
server_key_version: "encryptionkey@0"
I20260806 09:05:08.465185 5879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:05:08.469338 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:08.471032 5879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:05:08.471199 5879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "396ab2b6a705468dbd0c21b8d16a361b"
format_stamp: "Formatted at 2026-08-06 09:05:08 on dist-test-slave-8tdl"
server_key: "83a7022094eebc52b504e1d9a71437db"
server_key_iv: "db5d0d45864f3f24afcecfb49e60bd1c"
server_key_version: "encryptionkey@0"
I20260806 09:05:08.471504 5879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:08.495241 5879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:08.496243 5879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:08.496600 5879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:08.497668 5879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:08.499455 5879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:08.499550 5879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:08.499750 5879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:08.499836 5879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:08.536829 5879 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33079
I20260806 09:05:08.536924 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33079 every 8 connection(s)
I20260806 09:05:08.538388 5879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:05:08.539137 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 5879
I20260806 09:05:08.539327 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:05:08.539949 28398 external_mini_cluster.cc:1428] Setting key a98d280abec496789f2ecbf38d3e1df1
I20260806 09:05:08.542255 5744 heartbeater.cc:503] Master 127.27.187.190:36265 was elected leader, sending a full tablet report...
I20260806 09:05:08.552549 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.541304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49101 (local address 127.27.187.190:36265)
0806 09:05:08.541533 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.541540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.543334 (+ 1794us) server_negotiation.cc:407] Connection header received
0806 09:05:08.545498 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.545504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.545597 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.545711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.547034 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.547788 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.549460 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.549865 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.550004 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.551646 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:08.551651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:08.551654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:08.551675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:08.551715 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:08.551731 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:08.551837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:08.552092 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 09:05:08.552211 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:05:08.553594 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.541970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:36265 (local address 127.27.187.131:49101)
0806 09:05:08.543171 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:08.543216 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:08.545217 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:08.545972 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:08.545981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:08.546122 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:08.546860 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:08.546880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.547958 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.547963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:08.549293 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:08.549306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.550101 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.550108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:08.550421 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.551285 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:08.551288 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:08.551330 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:08.551413 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:08.551846 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:08.551858 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:08.551868 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:05:08.552063 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 09:05:08.552484 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":412,"threads_started":1}
I20260806 09:05:08.555289 6007 heartbeater.cc:347] Connected to a master server at 127.27.187.190:36265
I20260806 09:05:08.555708 6007 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:08.556628 6007 heartbeater.cc:511] Master 127.27.187.190:36265 requested a full tablet report, sending...
I20260806 09:05:08.558290 5557 ts_manager.cc:194] Registered new tserver with Master: 396ab2b6a705468dbd0c21b8d16a361b (127.27.187.131:33079)
I20260806 09:05:08.559303 5557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.131:49101
I20260806 09:05:08.569973 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:05:08.577600 5877 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:08.575356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.27.187.190:36265)
0806 09:05:08.575629 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.575679 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.576084 (+ 405us) server_negotiation.cc:407] Connection header received
0806 09:05:08.576360 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.576367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.576462 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.576600 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.577357 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37130: BlockingRecv error: recv got EOF from 127.0.0.1:37130 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:05:08.578013 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5617
I20260806 09:05:08.587121 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.579555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.27.187.129:39645)
0806 09:05:08.580203 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.580210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.580238 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:05:08.580300 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.580306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.580539 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.580730 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.581199 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.582215 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.583613 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.584207 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.584360 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.585570 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:08.585579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:08.585618 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:08.586213 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:08.586264 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:08.586284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:08.586419 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:08.586583 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 09:05:08.586759 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":290,"threads_started":1}
I20260806 09:05:08.588630 5739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:08.675560 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5617
I20260806 09:05:08.695683 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5748
I20260806 09:05:08.704093 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.696696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.27.187.130:35089)
0806 09:05:08.697246 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.697254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.697282 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:05:08.697369 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.697376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.697617 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.697798 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.698238 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.699335 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.700585 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.701179 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.701337 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.702483 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:08.702490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:08.702505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:08.703209 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:08.703272 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:08.703294 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:08.703422 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:08.703594 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 09:05:08.703742 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260806 09:05:08.705603 5870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:08.791076 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5748
I20260806 09:05:08.809810 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5879
I20260806 09:05:08.820199 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:08.811010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.27.187.131:33079)
0806 09:05:08.811846 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:08.811853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:08.811899 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:05:08.811982 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:08.811989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:08.812318 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:08.812534 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:08.813035 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.814107 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.815495 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:08.816281 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:08.816431 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:08.817778 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:08.817792 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:08.817806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:08.818862 (+ 1056us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:08.818936 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:08.818983 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:08.819134 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:08.819583 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 09:05:08.819797 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":514,"threads_started":1}
I20260806 09:05:08.821995 6002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:08.909480 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5879
I20260806 09:05:08.928704 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 5527
I20260806 09:05:08.930243 5588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:09.044006 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 5527
2026-08-06T09:05:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2726 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T09:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:05:09Z Disabled control of system clock
I20260806 09:05:09.137295 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38589
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:05:09.503176 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:09.503532 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:09.503623 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:09.511575 6026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:05:09.511713 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:09.511751 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:09.511803 6026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:05:09.511848 6026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:05:09.522195 6026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:09.523654 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:09.525556 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:09.539072 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:09.539176 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:09.539269 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:09.540462 6026 server_base.cc:1034] running on GCE node
I20260806 09:05:09.541442 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:09.543263 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:09.544557 6026 hybrid_clock.cc:648] HybridClock initialized: now 1786007109544515 us; error 59 us; skew 500 ppm
I20260806 09:05:09.544981 6026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:09.547824 6026 webserver.cc:460] Webserver started at http://127.27.187.190:43089/ using document root <none> and password file <none>
I20260806 09:05:09.548636 6026 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:09.548781 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:09.549153 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:09.552464 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "9176af1f424f43849340526b0a15e034"
format_stamp: "Formatted at 2026-08-06 09:05:09 on dist-test-slave-8tdl"
server_key: "7450e78af0ed97880bff22d453879efc"
server_key_iv: "b0a661bb17928e90abdd7256e4535e6b"
server_key_version: "encryptionkey@0"
I20260806 09:05:09.553395 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "9176af1f424f43849340526b0a15e034"
format_stamp: "Formatted at 2026-08-06 09:05:09 on dist-test-slave-8tdl"
server_key: "7450e78af0ed97880bff22d453879efc"
server_key_iv: "b0a661bb17928e90abdd7256e4535e6b"
server_key_version: "encryptionkey@0"
I20260806 09:05:09.559338 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:05:09.563515 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:09.565384 6026 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:05:09.565656 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "9176af1f424f43849340526b0a15e034"
format_stamp: "Formatted at 2026-08-06 09:05:09 on dist-test-slave-8tdl"
server_key: "7450e78af0ed97880bff22d453879efc"
server_key_iv: "b0a661bb17928e90abdd7256e4535e6b"
server_key_version: "encryptionkey@0"
I20260806 09:05:09.565956 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:09.572878 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:09.574028 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:09.574422 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:09.596153 6026 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38589
I20260806 09:05:09.596253 6091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38589 every 8 connection(s)
I20260806 09:05:09.597791 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:05:09.598523 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6026
I20260806 09:05:09.598825 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:05:09.599686 28398 external_mini_cluster.cc:1428] Setting key 5e7acda0dac7bda221d508fe79adb4d6
I20260806 09:05:09.603183 6092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:09.609498 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:09.601939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.27.187.190:38589)
0806 09:05:09.603178 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:09.603300 (+ 122us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:09.603427 (+ 127us) server_negotiation.cc:407] Connection header received
0806 09:05:09.605275 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:09.605307 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:09.605761 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:09.606288 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:09.607548 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:09.607555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:09.607569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:09.607584 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:09.607683 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:09.607702 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:09.607980 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:09.608148 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 09:05:09.608458 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":438,"threads_started":1}
I20260806 09:05:09.616266 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034: Bootstrap starting.
I20260806 09:05:09.619813 6092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:09.621074 6092 log.cc:824] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:09.624747 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034: No bootstrap required, opened a new log
I20260806 09:05:09.632839 6092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } }
I20260806 09:05:09.633322 6092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:09.633446 6092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9176af1f424f43849340526b0a15e034, State: Initialized, Role: FOLLOWER
I20260806 09:05:09.634150 6092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } }
I20260806 09:05:09.634471 6092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:05:09.634616 6092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:05:09.634907 6092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:09.637224 6092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } }
I20260806 09:05:09.637809 6092 leader_election.cc:302] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9176af1f424f43849340526b0a15e034; no voters:
I20260806 09:05:09.638478 6092 leader_election.cc:288] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:05:09.638643 6097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:05:09.639546 6097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [term 1 LEADER]: Becoming Leader. State: Replica: 9176af1f424f43849340526b0a15e034, State: Running, Role: LEADER
I20260806 09:05:09.640151 6097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } }
I20260806 09:05:09.640988 6092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:05:09.643479 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9176af1f424f43849340526b0a15e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } } }
I20260806 09:05:09.644016 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:09.645313 6099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9176af1f424f43849340526b0a15e034. Latest consensus state: current_term: 1 leader_uuid: "9176af1f424f43849340526b0a15e034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9176af1f424f43849340526b0a15e034" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38589 } } }
I20260806 09:05:09.645834 6099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:09.647189 6105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:05:09.653620 6105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:05:09.663324 6105 catalog_manager.cc:1347] Generated new cluster ID: b93773b08328451a9bd1648cc325beeb
I20260806 09:05:09.663457 6105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:05:09.676478 6105 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:05:09.677007 6105 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:05:09.685628 6105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9176af1f424f43849340526b0a15e034: Generated new TSK 0
I20260806 09:05:09.686656 6105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:05:09.697830 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--builtin_ntp_servers=127.27.187.148:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:05:10.073001 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:10.073330 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:10.073491 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:10.081409 6116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:10.081529 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:10.081707 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:05:10.091526 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:10.093012 6116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:10.095074 6116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:10.109747 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:10.109992 6121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:10.109750 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:10.110805 6116 server_base.cc:1034] running on GCE node
I20260806 09:05:10.111518 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:10.112601 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:10.113878 6116 hybrid_clock.cc:648] HybridClock initialized: now 1786007110113811 us; error 57 us; skew 500 ppm
I20260806 09:05:10.114305 6116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:10.121706 6116 webserver.cc:460] Webserver started at http://127.27.187.129:36051/ using document root <none> and password file <none>
I20260806 09:05:10.122730 6116 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:10.122905 6116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:10.123302 6116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:10.126400 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "96874778b9364facbe0bbc4a55928474"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "1da01406295bf209ccb37adc0d6f1c15"
server_key_iv: "70d3967acf0bf338bf28b8695d966d52"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.127278 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "96874778b9364facbe0bbc4a55928474"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "1da01406295bf209ccb37adc0d6f1c15"
server_key_iv: "70d3967acf0bf338bf28b8695d966d52"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.133308 6116 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:05:10.137773 6130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.139722 6116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:05:10.139955 6116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "96874778b9364facbe0bbc4a55928474"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "1da01406295bf209ccb37adc0d6f1c15"
server_key_iv: "70d3967acf0bf338bf28b8695d966d52"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.140240 6116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:10.146636 6116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:10.147665 6116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:10.147981 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:10.149159 6116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:10.150871 6116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:10.151000 6116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.151149 6116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:10.151242 6116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.189774 6116 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:33767
I20260806 09:05:10.189827 6242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:33767 every 8 connection(s)
I20260806 09:05:10.191354 6116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:05:10.193645 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6116
I20260806 09:05:10.193902 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:05:10.194824 28398 external_mini_cluster.cc:1428] Setting key 378a3e2c0371d823e69950f62745363f
I20260806 09:05:10.198417 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--builtin_ntp_servers=127.27.187.148:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:05:10.202608 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:10.194798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:33593 (local address 127.27.187.190:38589)
0806 09:05:10.195441 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:10.195446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:10.197246 (+ 1800us) server_negotiation.cc:407] Connection header received
0806 09:05:10.199594 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:10.199603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:10.199692 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:10.199828 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:10.201657 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:10.201666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:10.201673 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:10.201713 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:10.201776 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:10.201796 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:10.201954 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:10.202130 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 09:05:10.202208 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":402,"threads_started":1}
I20260806 09:05:10.203630 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:10.195561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38589 (local address 127.27.187.129:33593)
0806 09:05:10.197004 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:10.197075 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:10.199251 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:10.200181 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:10.200263 (+ 82us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:10.200386 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:10.201264 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:10.201266 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:10.201312 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:10.201419 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:10.201914 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:10.201923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:10.201929 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:05:10.202171 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 09:05:10.202456 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":451,"threads_started":1}
I20260806 09:05:10.205716 6243 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38589
I20260806 09:05:10.206182 6243 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:10.206961 6243 heartbeater.cc:511] Master 127.27.187.190:38589 requested a full tablet report, sending...
I20260806 09:05:10.209815 6057 ts_manager.cc:194] Registered new tserver with Master: 96874778b9364facbe0bbc4a55928474 (127.27.187.129:33767)
W20260806 09:05:10.568305 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:10.568629 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:10.568730 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:10.576568 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:10.576704 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:10.576821 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:05:10.587251 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:10.588654 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:10.590512 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:10.606364 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:10.607003 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:10.607055 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:10.608151 6247 server_base.cc:1034] running on GCE node
I20260806 09:05:10.608891 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:10.610256 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:10.611515 6247 hybrid_clock.cc:648] HybridClock initialized: now 1786007110611375 us; error 136 us; skew 500 ppm
I20260806 09:05:10.611913 6247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:10.615089 6247 webserver.cc:460] Webserver started at http://127.27.187.130:45223/ using document root <none> and password file <none>
I20260806 09:05:10.616036 6247 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:10.616154 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:10.616537 6247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:10.620263 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "d183843a5f0b49ca8437a0fee40ba51b"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "bda3b53a5cbb734e86e4e36dfb277549"
server_key_iv: "b4b9aa95737a8957ea0364415d376775"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.621161 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "d183843a5f0b49ca8437a0fee40ba51b"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "bda3b53a5cbb734e86e4e36dfb277549"
server_key_iv: "b4b9aa95737a8957ea0364415d376775"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.627188 6247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:05:10.631199 6261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.632953 6247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:05:10.633200 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "d183843a5f0b49ca8437a0fee40ba51b"
format_stamp: "Formatted at 2026-08-06 09:05:10 on dist-test-slave-8tdl"
server_key: "bda3b53a5cbb734e86e4e36dfb277549"
server_key_iv: "b4b9aa95737a8957ea0364415d376775"
server_key_version: "encryptionkey@0"
I20260806 09:05:10.633500 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:10.640611 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:10.641754 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:10.642113 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:10.643467 6247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:10.645274 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:10.645430 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.645603 6247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:10.645720 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:10.684489 6247 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38827
I20260806 09:05:10.684584 6373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38827 every 8 connection(s)
I20260806 09:05:10.686018 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:05:10.692670 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6247
I20260806 09:05:10.692891 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:05:10.693495 28398 external_mini_cluster.cc:1428] Setting key 97899f1076915964accec947d10d5f63
I20260806 09:05:10.695755 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:10.688931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:34047 (local address 127.27.187.190:38589)
0806 09:05:10.689173 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:10.689180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:10.690721 (+ 1541us) server_negotiation.cc:407] Connection header received
0806 09:05:10.692965 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:10.692970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:10.693058 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:10.693207 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:10.694976 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:10.694982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:10.694988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:10.695009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:10.695048 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:10.695063 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:10.695270 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:10.695353 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 09:05:10.695416 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:05:10.697124 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:10.689484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38589 (local address 127.27.187.130:34047)
0806 09:05:10.690508 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:10.690551 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:10.692536 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:10.693470 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:10.693485 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:10.693640 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:10.694577 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:10.694581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:10.694640 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:10.694787 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:10.695173 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:10.695181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:10.695187 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:05:10.695418 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 09:05:10.695700 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":334,"threads_started":1}
I20260806 09:05:10.697291 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--builtin_ntp_servers=127.27.187.148:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:05:10.699020 6374 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38589
I20260806 09:05:10.699368 6374 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:10.700064 6374 heartbeater.cc:511] Master 127.27.187.190:38589 requested a full tablet report, sending...
I20260806 09:05:10.701723 6057 ts_manager.cc:194] Registered new tserver with Master: d183843a5f0b49ca8437a0fee40ba51b (127.27.187.130:38827)
W20260806 09:05:11.073168 6377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:11.073508 6377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:11.073619 6377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:11.082017 6377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:11.082154 6377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:11.082276 6377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:05:11.092075 6377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:11.093412 6377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:11.095312 6377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:11.110158 6382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:11.110284 6383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:11.110152 6385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:11.111554 6377 server_base.cc:1034] running on GCE node
I20260806 09:05:11.112308 6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:11.113402 6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:11.114774 6377 hybrid_clock.cc:648] HybridClock initialized: now 1786007111114628 us; error 59 us; skew 500 ppm
I20260806 09:05:11.115185 6377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:05:11.117936 6377 webserver.cc:460] Webserver started at http://127.27.187.131:37347/ using document root <none> and password file <none>
I20260806 09:05:11.118813 6377 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:11.118963 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:11.119333 6377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:11.122495 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "93adbdccee7c419ab728d5798b5f83e7"
format_stamp: "Formatted at 2026-08-06 09:05:11 on dist-test-slave-8tdl"
server_key: "6d1fcb300148920550ec094b9c64624a"
server_key_iv: "d10b2108de7d0fba16565b695ab8749e"
server_key_version: "encryptionkey@0"
I20260806 09:05:11.123520 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "93adbdccee7c419ab728d5798b5f83e7"
format_stamp: "Formatted at 2026-08-06 09:05:11 on dist-test-slave-8tdl"
server_key: "6d1fcb300148920550ec094b9c64624a"
server_key_iv: "d10b2108de7d0fba16565b695ab8749e"
server_key_version: "encryptionkey@0"
I20260806 09:05:11.129671 6377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:05:11.133929 6392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:11.135782 6377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:05:11.135993 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "93adbdccee7c419ab728d5798b5f83e7"
format_stamp: "Formatted at 2026-08-06 09:05:11 on dist-test-slave-8tdl"
server_key: "6d1fcb300148920550ec094b9c64624a"
server_key_iv: "d10b2108de7d0fba16565b695ab8749e"
server_key_version: "encryptionkey@0"
I20260806 09:05:11.136271 6377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:11.143128 6377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:11.144122 6377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:11.144429 6377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:11.145565 6377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:11.147236 6377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:11.147365 6377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:11.147507 6377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:11.147614 6377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:11.183446 6377 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:43275
I20260806 09:05:11.183550 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:43275 every 8 connection(s)
I20260806 09:05:11.184922 6377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:05:11.190371 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6377
I20260806 09:05:11.190559 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:05:11.191177 28398 external_mini_cluster.cc:1428] Setting key 4735e11a2b62b82f7ac62361b64e4860
I20260806 09:05:11.195591 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:11.188074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:45739 (local address 127.27.187.190:38589)
0806 09:05:11.188338 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:11.188343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:11.190164 (+ 1821us) server_negotiation.cc:407] Connection header received
0806 09:05:11.192514 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:11.192520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:11.192585 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:11.192715 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:11.194770 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:11.194776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:11.194780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:11.194803 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:11.194844 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:11.194859 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:11.194997 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:11.195306 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 09:05:11.195366 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:05:11.196552 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:11.188594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38589 (local address 127.27.187.131:45739)
0806 09:05:11.189938 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:11.190002 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:11.192173 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:11.193167 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:11.193192 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:11.193323 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:11.194264 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:05:11.194269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:11.194328 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:05:11.194459 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:11.194993 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:11.195007 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:11.195017 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:05:11.195288 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 09:05:11.195508 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":407,"threads_started":1}
I20260806 09:05:11.198210 6506 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38589
I20260806 09:05:11.198518 6506 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:11.199241 6506 heartbeater.cc:511] Master 127.27.187.190:38589 requested a full tablet report, sending...
I20260806 09:05:11.200919 6057 ts_manager.cc:194] Registered new tserver with Master: 93adbdccee7c419ab728d5798b5f83e7 (127.27.187.131:43275)
I20260806 09:05:11.208078 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:05:11.213589 6243 heartbeater.cc:503] Master 127.27.187.190:38589 was elected leader, sending a full tablet report...
W20260806 09:05:11.215759 6245 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:05:11.214153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.27.187.190:38589)
0806 09:05:11.214343 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:11.214352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:11.214472 (+ 120us) server_negotiation.cc:407] Connection header received
0806 09:05:11.214731 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:11.214741 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:11.214799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:11.214918 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:11.215350 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43716: BlockingRecv error: recv got EOF from 127.0.0.1:43716 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:05:11.216056 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6116
I20260806 09:05:11.220235 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:11.217252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.27.187.129:33767)
0806 09:05:11.217928 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:11.217936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:11.217966 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:05:11.218034 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:11.218042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:11.218269 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:11.218415 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:11.219538 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:11.219545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:11.219559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:11.219573 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:11.219617 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:11.219633 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:11.219783 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:11.219838 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 09:05:11.219926 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":340,"threads_started":1}
I20260806 09:05:11.221554 6238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:11.310041 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6116
I20260806 09:05:11.329198 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6247
I20260806 09:05:11.333459 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:11.330240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.27.187.130:38827)
0806 09:05:11.330854 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:11.330863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:11.330899 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:05:11.330965 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:11.330972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:11.331245 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:11.331586 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:11.332687 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:11.332694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:11.332705 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:11.332724 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:11.332767 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:11.332782 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:11.332937 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:11.333070 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 09:05:11.333183 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260806 09:05:11.334944 6369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:11.421411 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6247
I20260806 09:05:11.439332 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6377
I20260806 09:05:11.444042 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:11.440443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.27.187.131:43275)
0806 09:05:11.441344 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:11.441351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:11.441383 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:05:11.441450 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:11.441456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:11.441706 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:11.441908 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:11.443203 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:11.443210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:11.443222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:05:11.443239 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:11.443290 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:05:11.443304 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:11.443501 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:11.443587 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 09:05:11.443699 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":491,"threads_started":1}
I20260806 09:05:11.445523 6501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:11.533948 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6377
I20260806 09:05:11.553805 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6026
I20260806 09:05:11.555235 6087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:11.667238 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6026
2026-08-06T09:05:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2619 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:05:11 dist-test-slave-8tdl krb5kdc[6524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:05:11 dist-test-slave-8tdl krb5kdc[6524](info): set up 2 sockets
Aug 06 09:05:11 dist-test-slave-8tdl krb5kdc[6524](info): commencing operation
krb5kdc: starting...
W20260806 09:05:12.840058 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.005s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:05:13 dist-test-slave-8tdl krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:05:13.168151 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:46369
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:05:13.528066 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:13.528403 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:13.528563 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:13.536406 6537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:05:13.536547 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:13.536638 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:13.536725 6537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:05:13.536795 6537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:05:13.547399 6537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:13.548894 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:13.550280 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:13.563731 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:13.563678 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:13.563911 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:13.564571 6537 server_base.cc:1034] running on GCE node
I20260806 09:05:13.565454 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:13.567219 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:13.568477 6537 hybrid_clock.cc:648] HybridClock initialized: now 1786007113568435 us; error 39 us; skew 500 ppm
I20260806 09:05:13.568850 6537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:05:13 dist-test-slave-8tdl krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:05:13.572813 6537 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:05:13.574779 6537 webserver.cc:460] Webserver started at http://127.27.187.190:46505/ using document root <none> and password file <none>
I20260806 09:05:13.575552 6537 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:13.575735 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:13.576076 6537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:13.579510 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "988233eb8c48459aaba014f0b841f517"
format_stamp: "Formatted at 2026-08-06 09:05:13 on dist-test-slave-8tdl"
server_key: "402c192d9215e2d29735ad9f6d6b34f3"
server_key_iv: "36e1c25a2a7066440a24bca8afa1a148"
server_key_version: "encryptionkey@0"
I20260806 09:05:13.580440 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "988233eb8c48459aaba014f0b841f517"
format_stamp: "Formatted at 2026-08-06 09:05:13 on dist-test-slave-8tdl"
server_key: "402c192d9215e2d29735ad9f6d6b34f3"
server_key_iv: "36e1c25a2a7066440a24bca8afa1a148"
server_key_version: "encryptionkey@0"
I20260806 09:05:13.586638 6537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:05:13.590952 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:13.592878 6537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:05:13.593150 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "988233eb8c48459aaba014f0b841f517"
format_stamp: "Formatted at 2026-08-06 09:05:13 on dist-test-slave-8tdl"
server_key: "402c192d9215e2d29735ad9f6d6b34f3"
server_key_iv: "36e1c25a2a7066440a24bca8afa1a148"
server_key_version: "encryptionkey@0"
I20260806 09:05:13.593411 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:13.617059 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:13.622756 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:13.623142 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:13.645725 6537 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:46369
I20260806 09:05:13.645761 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:46369 every 8 connection(s)
I20260806 09:05:13.647414 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:05:13.652146 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6537
I20260806 09:05:13.652458 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:05:13.653218 28398 external_mini_cluster.cc:1428] Setting key 6a063307b83fc8f8bd1f87b547411ed9
I20260806 09:05:13.653214 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:13.672811 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: Bootstrap starting.
Aug 06 09:05:13 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:05:13.677189 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:13.678875 6604 log.cc:824] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:13.683147 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: No bootstrap required, opened a new log
I20260806 09:05:13.685250 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:13.655741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.27.187.190:46369)
0806 09:05:13.657734 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:13.657748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:13.657797 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:05:13.660841 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:13.660880 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:13.661599 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:13.662192 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:13.664595 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:13.666857 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:13.669060 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:13.670405 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:13.670644 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:13.674985 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:13.675000 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:13.675023 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:13.675099 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:13.679660 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:13.680798 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:13.680809 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:13.680822 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:13.681059 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:13.681602 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:13.681607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:13.681610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:13.682403 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:13.682746 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:13.683248 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 09:05:13.683790 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1713,"thread_start_us":1425,"threads_started":1}
I20260806 09:05:13.692531 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } }
I20260806 09:05:13.692917 6604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:13.692996 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988233eb8c48459aaba014f0b841f517, State: Initialized, Role: FOLLOWER
I20260806 09:05:13.693511 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } }
I20260806 09:05:13.693728 6604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:05:13.693830 6604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:05:13.693960 6604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:13.696300 6604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } }
I20260806 09:05:13.696794 6604 leader_election.cc:302] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 988233eb8c48459aaba014f0b841f517; no voters:
I20260806 09:05:13.697535 6604 leader_election.cc:288] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:05:13.697708 6609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:05:13.698549 6609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [term 1 LEADER]: Becoming Leader. State: Replica: 988233eb8c48459aaba014f0b841f517, State: Running, Role: LEADER
I20260806 09:05:13.699705 6609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } }
I20260806 09:05:13.700103 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:05:13.702790 6610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "988233eb8c48459aaba014f0b841f517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } } }
I20260806 09:05:13.703032 6611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988233eb8c48459aaba014f0b841f517. Latest consensus state: current_term: 1 leader_uuid: "988233eb8c48459aaba014f0b841f517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "988233eb8c48459aaba014f0b841f517" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 46369 } } }
I20260806 09:05:13.703377 6610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:13.703457 6611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517 [sys.catalog]: This master's current role is: LEADER
I20260806 09:05:13.704308 6619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:05:13.710484 6619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:05:13.717533 6627 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:05:13.721342 6619 catalog_manager.cc:1347] Generated new cluster ID: 7031742556784ad88213d1fed857fc5d
I20260806 09:05:13.721491 6619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:05:13.744010 6619 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:05:13.745609 6619 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:05:13.756263 6619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: Generated new TSK 0
I20260806 09:05:13.757535 6619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:05:13.856104 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:05:14.248340 6631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:14.248644 6631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:14.248812 6631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:14.256765 6631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:14.256886 6631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:14.257059 6631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:05:14.267642 6631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:14.269043 6631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:14.270494 6631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:14.287433 6637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:14.287750 6636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:14.288008 6639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:14.289338 6631 server_base.cc:1034] running on GCE node
I20260806 09:05:14.290249 6631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:14.291674 6631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:14.292980 6631 hybrid_clock.cc:648] HybridClock initialized: now 1786007114292903 us; error 63 us; skew 500 ppm
I20260806 09:05:14.293398 6631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:05:14 dist-test-slave-8tdl krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:05:14.297706 6631 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:05:14.299877 6631 webserver.cc:460] Webserver started at http://127.27.187.129:34973/ using document root <none> and password file <none>
I20260806 09:05:14.300956 6631 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:14.301097 6631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:14.301492 6631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:14.305979 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "d612fef23e804a469c7b604990c0aeb4"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "cd02308f1f161c7fe7175a93e952c004"
server_key_iv: "b2fe3c41ebdf97b8a74cb90253553def"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.307183 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "d612fef23e804a469c7b604990c0aeb4"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "cd02308f1f161c7fe7175a93e952c004"
server_key_iv: "b2fe3c41ebdf97b8a74cb90253553def"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.314524 6631 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 09:05:14.319550 6646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.321605 6631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:05:14.321831 6631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "d612fef23e804a469c7b604990c0aeb4"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "cd02308f1f161c7fe7175a93e952c004"
server_key_iv: "b2fe3c41ebdf97b8a74cb90253553def"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.322104 6631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:14.355423 6631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:14.360347 6631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:14.360684 6631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:14.361768 6631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:14.363318 6631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:14.363422 6631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.363580 6631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:14.363660 6631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.399394 6631 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:42403
I20260806 09:05:14.399442 6759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:42403 every 8 connection(s)
I20260806 09:05:14.400938 6631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:05:14.405367 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6631
I20260806 09:05:14.405663 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:05:14.406497 28398 external_mini_cluster.cc:1428] Setting key e7281aa5353c3655cd3d70b9c378ea2e
Aug 06 09:05:14 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:05:14.425662 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:14.404329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:39121 (local address 127.27.187.190:46369)
0806 09:05:14.404867 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:14.404872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:14.406096 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 09:05:14.408192 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:14.408198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:14.408295 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:14.408419 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:14.410038 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.410879 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.412561 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.412965 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.413110 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:14.417628 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:14.417635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:14.417639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:14.417686 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:14.421432 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:14.422439 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:14.422450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:14.422458 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:14.422661 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:14.423215 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:14.423225 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:14.423232 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:14.423695 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:14.423878 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:14.424719 (+ 841us) server_negotiation.cc:299] Negotiation successful
0806 09:05:14.425038 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
I20260806 09:05:14.426407 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:14.404975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:46369 (local address 127.27.187.129:39121)
0806 09:05:14.405920 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:14.405972 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:14.407893 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:14.408686 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:14.408695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:14.409238 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:14.409843 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:14.409862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.411050 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.411054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:14.412388 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:14.412401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.413210 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.413218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:14.413483 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:14.414278 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:14.414314 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:14.417307 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:14.421662 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:14.421672 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:14.421697 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:14.422238 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:14.422848 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:14.422852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:14.422855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:14.423074 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:14.423873 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:14.423880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:14.424272 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 09:05:14.424527 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 09:05:14.424981 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":402,"threads_started":1}
I20260806 09:05:14.428563 6760 heartbeater.cc:347] Connected to a master server at 127.27.187.190:46369
I20260806 09:05:14.428936 6760 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:14.429816 6760 heartbeater.cc:511] Master 127.27.187.190:46369 requested a full tablet report, sending...
I20260806 09:05:14.432588 6569 ts_manager.cc:194] Registered new tserver with Master: d612fef23e804a469c7b604990c0aeb4 (127.27.187.129:42403)
I20260806 09:05:14.434162 6569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:39121
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:05:14.481098 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:05:14.846146 6766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:14.846532 6766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:14.846784 6766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:05:14.854525 6766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:14.854743 6766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:14.854955 6766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:05:14.865147 6766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:14.866605 6766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:14.868063 6766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:14.882283 6771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:14.882283 6772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:14.882368 6774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:14.884313 6766 server_base.cc:1034] running on GCE node
I20260806 09:05:14.885073 6766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:14.886231 6766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:14.887511 6766 hybrid_clock.cc:648] HybridClock initialized: now 1786007114887448 us; error 52 us; skew 500 ppm
I20260806 09:05:14.887902 6766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:05:14 dist-test-slave-8tdl krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:05:14.891500 6766 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:05:14.893175 6766 webserver.cc:460] Webserver started at http://127.27.187.130:38199/ using document root <none> and password file <none>
I20260806 09:05:14.893896 6766 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:14.894001 6766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:14.894273 6766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:14.897555 6766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "ece4fcf525f7478a9d5914b70712d42f"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "c8644d72a56fb217c680f7ff9a4389b9"
server_key_iv: "a31db7c98e8658497de37d751c7422cd"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.898468 6766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "ece4fcf525f7478a9d5914b70712d42f"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "c8644d72a56fb217c680f7ff9a4389b9"
server_key_iv: "a31db7c98e8658497de37d751c7422cd"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.904546 6766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:05:14.908495 6781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.910348 6766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:05:14.910573 6766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ece4fcf525f7478a9d5914b70712d42f"
format_stamp: "Formatted at 2026-08-06 09:05:14 on dist-test-slave-8tdl"
server_key: "c8644d72a56fb217c680f7ff9a4389b9"
server_key_iv: "a31db7c98e8658497de37d751c7422cd"
server_key_version: "encryptionkey@0"
I20260806 09:05:14.910894 6766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:14.931089 6766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:14.936508 6766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:14.936868 6766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:14.938005 6766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:14.939599 6766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:14.939678 6766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.939864 6766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:14.939921 6766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:14.978701 6766 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36875
I20260806 09:05:14.978816 6893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36875 every 8 connection(s)
I20260806 09:05:14.980265 6766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:05:14.983879 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6766
I20260806 09:05:14.984117 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:05:14.984825 28398 external_mini_cluster.cc:1428] Setting key e24e67588f45983decaaddd5b069a393
Aug 06 09:05:14 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:05:15.010056 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:14.983533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:38537 (local address 127.27.187.190:46369)
0806 09:05:14.984132 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:14.984137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:14.985425 (+ 1288us) server_negotiation.cc:407] Connection header received
0806 09:05:14.987860 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:14.987865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:14.987939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:14.988041 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:14.990440 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.991380 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.994128 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.994784 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.994909 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.000415 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:15.000428 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:15.000436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:15.000503 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:15.005629 (+ 5126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.006847 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.006857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.006863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.007050 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.007615 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.007624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.007630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.008086 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:15.008251 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.009209 (+ 958us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.009534 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":332,"threads_started":1}
I20260806 09:05:15.010974 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:14.984218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:46369 (local address 127.27.187.130:38537)
0806 09:05:14.985227 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:14.985284 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:14.987505 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:14.988371 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:14.988386 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:14.989292 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:14.990205 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:14.990235 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.991637 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.991645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:14.993927 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:14.993947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:14.995055 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:14.995065 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:14.995427 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:14.996327 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:14.996371 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:14.999793 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:15.005934 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:15.005944 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:15.005961 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:15.006545 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:15.007221 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:15.007224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:15.007227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:15.007460 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:15.008284 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:15.008291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:15.008728 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 09:05:15.009017 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 09:05:15.009582 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":424,"threads_started":1}
I20260806 09:05:15.012817 6894 heartbeater.cc:347] Connected to a master server at 127.27.187.190:46369
I20260806 09:05:15.013221 6894 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:15.014351 6894 heartbeater.cc:511] Master 127.27.187.190:46369 requested a full tablet report, sending...
I20260806 09:05:15.016615 6569 ts_manager.cc:194] Registered new tserver with Master: ece4fcf525f7478a9d5914b70712d42f (127.27.187.130:36875)
I20260806 09:05:15.017665 6569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:38537
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:05:15.063544 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:05:15.434492 6900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:05:15.434885 6900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:05:15.434991 6900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:05:15.437775 6760 heartbeater.cc:503] Master 127.27.187.190:46369 was elected leader, sending a full tablet report...
W20260806 09:05:15.443396 6900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:05:15.443532 6900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:05:15.443728 6900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:05:15.454360 6900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:46369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:05:15.455850 6900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:05:15.457194 6900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:05:15.472360 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:15.472338 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:05:15.472391 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:05:15.472769 6900 server_base.cc:1034] running on GCE node
I20260806 09:05:15.473812 6900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:05:15.475154 6900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:05:15.476444 6900 hybrid_clock.cc:648] HybridClock initialized: now 1786007115476386 us; error 99 us; skew 500 ppm
I20260806 09:05:15.476837 6900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:05:15 dist-test-slave-8tdl krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:05:15.484721 6900 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:05:15.486755 6900 webserver.cc:460] Webserver started at http://127.27.187.131:42965/ using document root <none> and password file <none>
I20260806 09:05:15.487568 6900 fs_manager.cc:359] Metadata directory not provided
I20260806 09:05:15.487735 6900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:05:15.488085 6900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:05:15.491958 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "08a7030f789d41668bb61a74115133a1"
format_stamp: "Formatted at 2026-08-06 09:05:15 on dist-test-slave-8tdl"
server_key: "04777e5c20e803b2a0fed2bdc9823d06"
server_key_iv: "66d0455a8676d2eeb5092c08b53942de"
server_key_version: "encryptionkey@0"
I20260806 09:05:15.492980 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "08a7030f789d41668bb61a74115133a1"
format_stamp: "Formatted at 2026-08-06 09:05:15 on dist-test-slave-8tdl"
server_key: "04777e5c20e803b2a0fed2bdc9823d06"
server_key_iv: "66d0455a8676d2eeb5092c08b53942de"
server_key_version: "encryptionkey@0"
I20260806 09:05:15.499811 6900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:05:15.504498 6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:15.506433 6900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:05:15.506630 6900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "08a7030f789d41668bb61a74115133a1"
format_stamp: "Formatted at 2026-08-06 09:05:15 on dist-test-slave-8tdl"
server_key: "04777e5c20e803b2a0fed2bdc9823d06"
server_key_iv: "66d0455a8676d2eeb5092c08b53942de"
server_key_version: "encryptionkey@0"
I20260806 09:05:15.506927 6900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:05:15.560282 6900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:05:15.567852 6900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:05:15.568238 6900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:05:15.569654 6900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:05:15.571659 6900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:05:15.571768 6900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:15.571899 6900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:05:15.571969 6900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:05:15.623924 6900 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:39819
I20260806 09:05:15.623989 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:39819 every 8 connection(s)
I20260806 09:05:15.625953 6900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:05:15.630650 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 6900
I20260806 09:05:15.632023 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:05:15.634176 28398 external_mini_cluster.cc:1428] Setting key 2e5d54760ac229988ad4f897e3a8172c
Aug 06 09:05:15 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:05:15.676141 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.635562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:47779 (local address 127.27.187.190:46369)
0806 09:05:15.638017 (+ 2455us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:15.638051 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:15.642845 (+ 4794us) server_negotiation.cc:407] Connection header received
0806 09:05:15.645378 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:15.645413 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:15.645592 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:15.645803 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:15.648758 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.650003 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.653346 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.654211 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.654479 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.663372 (+ 8893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:15.663380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:15.663387 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:15.663437 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:15.669262 (+ 5825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.671125 (+ 1863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.671139 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.671142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.671541 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.672920 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.672932 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.672934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.673583 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:15.673774 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.675182 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.675617 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2028,"thread_start_us":1281,"threads_started":1}
I20260806 09:05:15.678550 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.639026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:46369 (local address 127.27.187.131:47779)
0806 09:05:15.640883 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:15.640980 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:15.644719 (+ 3739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:15.646332 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:15.646346 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:15.647409 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:05:15.648446 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.648473 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.650223 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.650228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.652943 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.652973 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.654753 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.654764 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.655337 (+ 573us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.656892 (+ 1555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:05:15.656935 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:05:15.662947 (+ 6012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:05:15.669598 (+ 6651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:15.669610 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:15.669639 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:15.670890 (+ 1251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:15.671815 (+ 925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:05:15.671821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:05:15.671825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:05:15.672715 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:05:15.673828 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:05:15.673838 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:05:15.674509 (+ 671us) client_negotiation.cc:769] Sending connection context
0806 09:05:15.674999 (+ 490us) client_negotiation.cc:240] Negotiation successful
0806 09:05:15.675901 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":895,"threads_started":1}
I20260806 09:05:15.681427 7031 heartbeater.cc:347] Connected to a master server at 127.27.187.190:46369
I20260806 09:05:15.681913 7031 heartbeater.cc:464] Registering TS with master...
I20260806 09:05:15.683377 7031 heartbeater.cc:511] Master 127.27.187.190:46369 requested a full tablet report, sending...
I20260806 09:05:15.686342 6569 ts_manager.cc:194] Registered new tserver with Master: 08a7030f789d41668bb61a74115133a1 (127.27.187.131:39819)
I20260806 09:05:15.687465 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:05:15.687909 6569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:47779
I20260806 09:05:15.721845 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.694664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.27.187.190:46369)
0806 09:05:15.695057 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:15.695071 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:15.695664 (+ 593us) server_negotiation.cc:407] Connection header received
0806 09:05:15.695872 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:15.695880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:15.695987 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:15.696102 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:15.698103 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.699190 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.701047 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.701917 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.702433 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.707207 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:15.707217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:15.707221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:15.707284 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:15.712579 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.714130 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.714139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.714158 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.714455 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:15.715334 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:15.715338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:15.715341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:15.716532 (+ 1191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:15.717114 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.719540 (+ 2426us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.720628 (+ 1088us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:05:15.732407 6569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:05:15.747619 6569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:05:15.802102 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.778442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36875 (local address 127.0.0.1:48344)
0806 09:05:15.779273 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:15.779374 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:15.779900 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:15.785752 (+ 5852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:15.785768 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:15.785836 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:05:15.787793 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.787812 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.792029 (+ 4217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.792047 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.797622 (+ 5575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.797657 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.800980 (+ 3323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.800986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.801218 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.801287 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 09:05:15.801412 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 09:05:15.801641 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":58,"thread_start_us":368,"threads_started":1}
I20260806 09:05:15.802636 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.778545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.27.187.130:36875)
0806 09:05:15.784142 (+ 5597us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:15.784192 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:15.784262 (+ 70us) server_negotiation.cc:407] Connection header received
0806 09:05:15.784499 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:15.784508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:15.785097 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:15.787295 (+ 2198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:05:15.788732 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.791637 (+ 2905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.797991 (+ 6354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.800539 (+ 2548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.800706 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.801756 (+ 1050us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.801867 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.802045 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5124,"thread_start_us":4978,"threads_started":1}
I20260806 09:05:15.803251 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.780377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42403 (local address 127.0.0.1:40526)
0806 09:05:15.782127 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:15.782144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:15.782318 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:15.787020 (+ 4702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:15.787044 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:15.787196 (+ 152us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:05:15.788771 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.788787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.795131 (+ 6344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.795182 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.799861 (+ 4679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.799897 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.802276 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.802283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.802448 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.802494 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:05:15.802560 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 09:05:15.802788 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":116,"thread_start_us":1362,"threads_started":1}
I20260806 09:05:15.804132 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.781532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.27.187.129:42403)
0806 09:05:15.785368 (+ 3836us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:15.785395 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:15.785474 (+ 79us) server_negotiation.cc:407] Connection header received
0806 09:05:15.785684 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:15.785696 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:15.786234 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:15.787740 (+ 1506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:05:15.789180 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.794649 (+ 5469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.800294 (+ 5645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.802043 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.802923 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.803125 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.803213 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.803535 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3491,"thread_start_us":3080,"threads_started":1}
I20260806 09:05:15.805226 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.779219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:39819 (local address 127.0.0.1:60324)
0806 09:05:15.781791 (+ 2572us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:15.781820 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:15.781977 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:15.786409 (+ 4432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:15.786426 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:15.786481 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:05:15.789029 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.789054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.794494 (+ 5440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.794505 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.799764 (+ 5259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:15.799795 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.802645 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.802651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:15.802871 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.802966 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 09:05:15.803055 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 09:05:15.803226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"mutex_wait_us":396,"thread_start_us":2081,"threads_started":1}
I20260806 09:05:15.805670 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:15.780306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.27.187.131:39819)
0806 09:05:15.783733 (+ 3427us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:15.783764 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:15.783847 (+ 83us) server_negotiation.cc:407] Connection header received
0806 09:05:15.784258 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:15.784269 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:15.785107 (+ 838us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:15.785682 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:05:15.789475 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.793608 (+ 4133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.800294 (+ 6686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:15.802043 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:15.802262 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:15.802444 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:15.803444 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0806 09:05:15.803805 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2785,"thread_start_us":2456,"threads_started":1}
I20260806 09:05:15.811007 6829 tablet_service.cc:1467] Processing CreateTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 (DEFAULT_TABLE table=test-table [id=ffd618af7b41497c8764c977d29b5624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:05:15.811445 6694 tablet_service.cc:1467] Processing CreateTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 (DEFAULT_TABLE table=test-table [id=ffd618af7b41497c8764c977d29b5624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:05:15.812150 6965 tablet_service.cc:1467] Processing CreateTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 (DEFAULT_TABLE table=test-table [id=ffd618af7b41497c8764c977d29b5624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:05:15.814340 6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a35b6166d4a09b3cec0de6b7f9472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:15.814386 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a35b6166d4a09b3cec0de6b7f9472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:15.814411 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828a35b6166d4a09b3cec0de6b7f9472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:05:15.846915 7050 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Bootstrap starting.
I20260806 09:05:15.857241 7052 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Bootstrap starting.
I20260806 09:05:15.857250 7051 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Bootstrap starting.
I20260806 09:05:15.865628 7050 tablet_bootstrap.cc:654] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:15.868943 7051 tablet_bootstrap.cc:654] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:15.869696 7052 tablet_bootstrap.cc:654] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Neither blocks nor log segments found. Creating new log.
I20260806 09:05:15.870383 7050 log.cc:824] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:15.871726 7051 log.cc:824] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:15.873829 7052 log.cc:824] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:05:15.882561 7050 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: No bootstrap required, opened a new log
I20260806 09:05:15.883285 7050 ts_tablet_manager.cc:1397] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.008s
I20260806 09:05:15.885463 7051 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: No bootstrap required, opened a new log
I20260806 09:05:15.886597 7051 ts_tablet_manager.cc:1397] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.006s
I20260806 09:05:15.887039 7052 tablet_bootstrap.cc:492] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: No bootstrap required, opened a new log
I20260806 09:05:15.888252 7052 ts_tablet_manager.cc:1397] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.013s
I20260806 09:05:15.912447 7050 raft_consensus.cc:348] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.913518 7050 raft_consensus.cc:374] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:15.913950 7050 raft_consensus.cc:729] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a7030f789d41668bb61a74115133a1, State: Initialized, Role: FOLLOWER
I20260806 09:05:15.915967 7050 consensus_queue.cc:260] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.925721 7031 heartbeater.cc:503] Master 127.27.187.190:46369 was elected leader, sending a full tablet report...
I20260806 09:05:15.925910 7052 raft_consensus.cc:348] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.927958 7052 raft_consensus.cc:374] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:15.928167 7052 raft_consensus.cc:729] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece4fcf525f7478a9d5914b70712d42f, State: Initialized, Role: FOLLOWER
I20260806 09:05:15.928768 7050 ts_tablet_manager.cc:1428] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Time spent starting tablet: real 0.045s user 0.026s sys 0.017s
I20260806 09:05:15.927594 7051 raft_consensus.cc:348] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.932528 7051 raft_consensus.cc:374] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:05:15.930572 7052 consensus_queue.cc:260] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.933076 7051 raft_consensus.cc:729] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d612fef23e804a469c7b604990c0aeb4, State: Initialized, Role: FOLLOWER
I20260806 09:05:15.935647 7051 consensus_queue.cc:260] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:15.937103 6894 heartbeater.cc:503] Master 127.27.187.190:46369 was elected leader, sending a full tablet report...
I20260806 09:05:15.938455 7052 ts_tablet_manager.cc:1428] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Time spent starting tablet: real 0.050s user 0.035s sys 0.011s
I20260806 09:05:15.943886 7051 ts_tablet_manager.cc:1428] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Time spent starting tablet: real 0.057s user 0.045s sys 0.007s
W20260806 09:05:15.983590 6895 tablet.cc:2369] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:05:16.039093 7058 raft_consensus.cc:482] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:05:16.039587 7058 raft_consensus.cc:504] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:16.043497 7058 leader_election.cc:288] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece4fcf525f7478a9d5914b70712d42f (127.27.187.130:36875), 08a7030f789d41668bb61a74115133a1 (127.27.187.131:39819)
I20260806 09:05:16.060431 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.045100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:39715 (local address 127.27.187.130:36875)
0806 09:05:16.045673 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.045695 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.047107 (+ 1412us) server_negotiation.cc:407] Connection header received
0806 09:05:16.047674 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.047693 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.047917 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.048149 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:05:16.050063 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.051821 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.057175 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.058552 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.058986 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.059176 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.059691 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.059912 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260806 09:05:16.060704 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.044408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36875 (local address 127.27.187.129:39715)
0806 09:05:16.046878 (+ 2470us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:16.046934 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:16.047240 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:16.048237 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:16.048245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:16.048298 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:05:16.049586 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:16.049612 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.052175 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.052187 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:16.056825 (+ 4638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:16.056865 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.059034 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.059044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:16.059355 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.059402 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 09:05:16.059644 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 09:05:16.059912 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2001,"mutex_wait_us":534,"thread_start_us":1611,"threads_started":1}
I20260806 09:05:16.060938 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.044937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:33663 (local address 127.27.187.131:39819)
0806 09:05:16.045553 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.045573 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.047256 (+ 1683us) server_negotiation.cc:407] Connection header received
0806 09:05:16.047673 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.047692 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.047916 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.048149 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:05:16.050124 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.051914 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.057393 (+ 5479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.059101 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.059530 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.059684 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.060231 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.060502 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260806 09:05:16.061455 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.044415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:39819 (local address 127.27.187.129:33663)
0806 09:05:16.047019 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0806 09:05:16.047039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:05:16.047331 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:05:16.048176 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:05:16.048181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:05:16.048226 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:05:16.049693 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:16.049729 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.052166 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.052185 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:16.056956 (+ 4771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:05:16.056984 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.059576 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.059586 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:05:16.059882 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.059928 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:05:16.060175 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 09:05:16.060476 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321,"mutex_wait_us":235,"spinlock_wait_cycles":44928,"thread_start_us":1209,"threads_started":1}
I20260806 09:05:16.063153 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a35b6166d4a09b3cec0de6b7f9472" candidate_uuid: "d612fef23e804a469c7b604990c0aeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a7030f789d41668bb61a74115133a1" is_pre_election: true
I20260806 09:05:16.063182 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a35b6166d4a09b3cec0de6b7f9472" candidate_uuid: "d612fef23e804a469c7b604990c0aeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece4fcf525f7478a9d5914b70712d42f" is_pre_election: true
I20260806 09:05:16.063956 6985 raft_consensus.cc:2454] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d612fef23e804a469c7b604990c0aeb4 in term 0.
I20260806 09:05:16.063973 6849 raft_consensus.cc:2454] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d612fef23e804a469c7b604990c0aeb4 in term 0.
I20260806 09:05:16.064980 6649 leader_election.cc:302] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a7030f789d41668bb61a74115133a1, d612fef23e804a469c7b604990c0aeb4; no voters:
I20260806 09:05:16.065708 7058 raft_consensus.cc:2781] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:05:16.065999 7058 raft_consensus.cc:482] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:05:16.066180 7058 raft_consensus.cc:3037] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:16.070230 7058 raft_consensus.cc:504] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:16.071353 7058 leader_election.cc:288] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [CANDIDATE]: Term 1 election: Requested vote from peers ece4fcf525f7478a9d5914b70712d42f (127.27.187.130:36875), 08a7030f789d41668bb61a74115133a1 (127.27.187.131:39819)
I20260806 09:05:16.072194 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a35b6166d4a09b3cec0de6b7f9472" candidate_uuid: "d612fef23e804a469c7b604990c0aeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece4fcf525f7478a9d5914b70712d42f"
I20260806 09:05:16.072216 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828a35b6166d4a09b3cec0de6b7f9472" candidate_uuid: "d612fef23e804a469c7b604990c0aeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a7030f789d41668bb61a74115133a1"
I20260806 09:05:16.072557 6985 raft_consensus.cc:3037] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:16.072559 6849 raft_consensus.cc:3037] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:05:16.076326 6849 raft_consensus.cc:2454] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d612fef23e804a469c7b604990c0aeb4 in term 1.
I20260806 09:05:16.076499 6985 raft_consensus.cc:2454] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d612fef23e804a469c7b604990c0aeb4 in term 1.
I20260806 09:05:16.077160 6647 leader_election.cc:302] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d612fef23e804a469c7b604990c0aeb4, ece4fcf525f7478a9d5914b70712d42f; no voters:
I20260806 09:05:16.077803 7058 raft_consensus.cc:2781] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:05:16.078649 7058 raft_consensus.cc:686] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 1 LEADER]: Becoming Leader. State: Replica: d612fef23e804a469c7b604990c0aeb4, State: Running, Role: LEADER
I20260806 09:05:16.079542 7058 consensus_queue.cc:237] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } }
I20260806 09:05:16.088220 6568 catalog_manager.cc:5495] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d612fef23e804a469c7b604990c0aeb4 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "d612fef23e804a469c7b604990c0aeb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d612fef23e804a469c7b604990c0aeb4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 } health_report { overall_health: UNKNOWN } } }
W20260806 09:05:16.130630 7032 tablet.cc:2369] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:05:16.135918 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.124276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.27.187.129:42403)
0806 09:05:16.124703 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.124713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.124873 (+ 160us) server_negotiation.cc:407] Connection header received
0806 09:05:16.125457 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.125470 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.125672 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.126029 (+ 357us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:05:16.127099 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.128324 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.130840 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.131849 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.132154 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.133343 (+ 1189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:05:16.134485 (+ 1142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:05:16.134844 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.135054 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.135247 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260806 09:05:16.152141 6985 raft_consensus.cc:1261] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 1 FOLLOWER]: Refusing update from remote peer d612fef23e804a469c7b604990c0aeb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:05:16.152201 6849 raft_consensus.cc:1261] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 1 FOLLOWER]: Refusing update from remote peer d612fef23e804a469c7b604990c0aeb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:05:16.154408 7061 consensus_queue.cc:1035] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece4fcf525f7478a9d5914b70712d42f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 09:05:16.155122 6761 tablet.cc:2369] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:05:16.155548 7058 consensus_queue.cc:1035] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a7030f789d41668bb61a74115133a1" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:05:16.216297 6569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:05:16.217157 6569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:05:16.228046 6569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 988233eb8c48459aaba014f0b841f517: Sending DeleteTablet for 3 replicas of tablet 828a35b6166d4a09b3cec0de6b7f9472
I20260806 09:05:16.232182 6829 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:05:16 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260806 09:05:16.232463 6965 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:05:16 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:60324
I20260806 09:05:16.235127 6694 tablet_service.cc:1514] Processing DeleteTablet for tablet 828a35b6166d4a09b3cec0de6b7f9472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:05:16 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:40526
I20260806 09:05:16.244468 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6631
I20260806 09:05:16.245064 7074 tablet_replica.cc:330] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: stopping tablet replica
I20260806 09:05:16.246778 7075 tablet_replica.cc:330] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: stopping tablet replica
I20260806 09:05:16.249469 7074 raft_consensus.cc:2229] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:05:16.252285 7075 raft_consensus.cc:2229] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:05:16.254587 7075 raft_consensus.cc:2258] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:05:16.255942 7076 tablet_replica.cc:330] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: stopping tablet replica
I20260806 09:05:16.256579 7074 raft_consensus.cc:2258] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:05:16.257705 7076 raft_consensus.cc:2229] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:05:16.263221 7076 raft_consensus.cc:2258] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:05:16.266646 7075 ts_tablet_manager.cc:1905] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:05:16.269649 7074 ts_tablet_manager.cc:1905] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:05:16.288545 7076 ts_tablet_manager.cc:1905] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:05:16.290812 7075 ts_tablet_manager.cc:1918] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:05:16.291230 7075 log.cc:1196] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/828a35b6166d4a09b3cec0de6b7f9472
I20260806 09:05:16.292476 7075 ts_tablet_manager.cc:1939] T 828a35b6166d4a09b3cec0de6b7f9472 P 08a7030f789d41668bb61a74115133a1: Deleting consensus metadata
I20260806 09:05:16.292747 7074 ts_tablet_manager.cc:1918] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:05:16.293169 7074 log.cc:1196] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/828a35b6166d4a09b3cec0de6b7f9472
I20260806 09:05:16.295379 7074 ts_tablet_manager.cc:1939] T 828a35b6166d4a09b3cec0de6b7f9472 P ece4fcf525f7478a9d5914b70712d42f: Deleting consensus metadata
I20260806 09:05:16.297760 6555 catalog_manager.cc:4841] TS 08a7030f789d41668bb61a74115133a1 (127.27.187.131:39819): tablet 828a35b6166d4a09b3cec0de6b7f9472 (table test-table [id=ffd618af7b41497c8764c977d29b5624]) successfully deleted
I20260806 09:05:16.301659 6553 catalog_manager.cc:4841] TS ece4fcf525f7478a9d5914b70712d42f (127.27.187.130:36875): tablet 828a35b6166d4a09b3cec0de6b7f9472 (table test-table [id=ffd618af7b41497c8764c977d29b5624]) successfully deleted
Aug 06 09:05:16 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:05:16.319532 7076 ts_tablet_manager.cc:1918] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:05:16.320304 7076 log.cc:1196] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/828a35b6166d4a09b3cec0de6b7f9472
I20260806 09:05:16.322623 7076 ts_tablet_manager.cc:1939] T 828a35b6166d4a09b3cec0de6b7f9472 P d612fef23e804a469c7b604990c0aeb4: Deleting consensus metadata
I20260806 09:05:16.328660 6553 catalog_manager.cc:4841] TS d612fef23e804a469c7b604990c0aeb4 (127.27.187.129:42403): tablet 828a35b6166d4a09b3cec0de6b7f9472 (table test-table [id=ffd618af7b41497c8764c977d29b5624]) successfully deleted
I20260806 09:05:16.337407 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.249049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.27.187.129:42403)
0806 09:05:16.249844 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.249863 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.261473 (+ 11610us) server_negotiation.cc:407] Connection header received
0806 09:05:16.263526 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.263565 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.263937 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.264192 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:16.273430 (+ 9238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.277958 (+ 4528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.290970 (+ 13012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.292385 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.292582 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.316012 (+ 23430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:16.316083 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:16.316164 (+ 81us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:16.316367 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:16.330056 (+ 13689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.332113 (+ 2057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.332129 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.332132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.332500 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.333476 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.333482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.333484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.334233 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:16.335033 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.336008 (+ 975us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.336747 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338}
I20260806 09:05:16.339311 6755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:16.515153 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6631
I20260806 09:05:16.544328 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6766
Aug 06 09:05:16 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:05:16.573832 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.545445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.27.187.130:36875)
0806 09:05:16.545774 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.545781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.545843 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:05:16.546132 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.546136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.546257 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.546397 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:16.548663 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.549891 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.552225 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.553020 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.553244 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.561474 (+ 8230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:16.561495 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:16.561527 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:16.561632 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:16.568236 (+ 6604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.569799 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.569810 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.569813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.570075 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.570927 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.570952 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.570955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.571685 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:16.572097 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.572747 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.573049 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 09:05:16.574901 6889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:16.734127 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6766
I20260806 09:05:16.764595 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6900
Aug 06 09:05:16 dist-test-slave-8tdl krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:05:16.795984 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:05:16.766002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.27.187.131:39819)
0806 09:05:16.767133 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0806 09:05:16.767148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:05:16.767203 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:05:16.767292 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:05:16.767297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:05:16.767416 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:05:16.767659 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:05:16.770292 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.772372 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.775683 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:05:16.776393 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:05:16.776673 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:05:16.785414 (+ 8741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:05:16.785448 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:05:16.785474 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:05:16.785544 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:05:16.790505 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.792074 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.792085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.792094 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.792344 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:05:16.793068 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:05:16.793083 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:05:16.793086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:05:16.793712 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:05:16.794341 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0806 09:05:16.794896 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 09:05:16.795325 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":558,"threads_started":1}
I20260806 09:05:16.797094 7026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:16.914628 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6900
I20260806 09:05:16.939757 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 6537
I20260806 09:05:16.941499 6599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:05:17.094837 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 6537
2026-08-06T09:05:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5445 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:05:17 dist-test-slave-8tdl krb5kdc[7086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:05:17 dist-test-slave-8tdl krb5kdc[7086](info): set up 2 sockets
Aug 06 09:05:17 dist-test-slave-8tdl krb5kdc[7086](info): commencing operation
krb5kdc: starting...
W20260806 09:05:18.330206 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:05:18 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:05:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangeradmin_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangeradmin_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangerlookup_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangerlookup_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
I20260806 09:05:18.792210 28398 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvYHTS_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/postgres -l logfile start
2026-08-06 09:05:21.911 UTC [7116] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:05:21.912 UTC [7116] LOG: listening on IPv4 address "127.27.187.148", port 44403
2026-08-06 09:05:21.917 UTC [7116] LOG: listening on Unix socket "/tmp/.s.PGSQL.44403"
2026-08-06 09:05:21.944 UTC [7118] LOG: database system was shut down at 2026-08-06 09:05:20 UTC
2026-08-06 09:05:21.948 UTC [7116] LOG: database system is ready to accept connections
I20260806 09:05:22.984474 28398 mini_postgres.cc:96] Postgres bound to 44403
2026-08-06 09:05:23.011 UTC [7127] FATAL: database "slave" does not exist
127.27.187.148:44403 - accepting connections
I20260806 09:05:23.013067 28398 mini_ranger.cc:161] Starting Ranger...
I20260806 09:05:23.050608 28398 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:05:23.698491 28398 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:05:23.698700 28398 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin
2026-08-06 09:05:24,061 [I] DB FLAVOR :POSTGRES
2026-08-06 09:05:24,062 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:05:24,062 [I] Checking connection..
2026-08-06 09:05:24,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:05:24,374 [I] Checking connection passed.
2026-08-06 09:05:24,374 [I] --------- Verifying version history table ---------
2026-08-06 09:05:24,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:05:24,635 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:05:24,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:05:24,914 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:05:24,914 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:05:24,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:05:25.170 UTC [7191] WARNING: there is no transaction in progress
2026-08-06 09:05:25,178 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:05:25,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:05:25,450 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:05:25,451 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:05:25,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:05:25,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:05:25,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-06 09:05:26,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:05:26,527 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:05:26,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:05:26,813 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:05:26,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:05:27,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:05:27,329 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:05:27,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:05:28.453 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.472 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.492 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.510 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.529 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.551 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.620 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.629 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.647 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.659 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.672 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.688 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.699 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.707 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.716 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:28.725 UTC [7302] WARNING: there is no transaction in progress
2026-08-06 09:05:29,595 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:05:29,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:05:29,857 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:05:29,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:05:30,141 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:05:30,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:05:30,416 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:05:30,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:05:30,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:05:30,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-06 09:05:31,212 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:05:31,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:05:31,479 [I] DB_PATCHES have already been applied
I20260806 09:05:31.485438 28398 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvYHTS_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:05:31.485539 28398 mini_ranger.cc:193] Using host: 127.27.187.148
I20260806 09:05:31.509888 28398 mini_ranger.cc:227] Ranger admin URL: http://127.27.187.148:34677
Aug 06, 2026 9:05:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:05:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:05:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:05:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:05:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:05:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:05:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:05:32.712085 28398 mini_ranger.cc:160] Time spent starting Ranger: real 9.699s user 0.003s sys 0.070s
Aug 06, 2026 9:05:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 9:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:05:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:05:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:05:57.788777 28398 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangerkms_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/rangerkms_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.187.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.187.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:05:58 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:05:58.043426 28398 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:05:58.076292 28398 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:05:59.277542 28398 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:05:59.277719 28398 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms
I20260806 09:05:59.277868 28398 mini_ranger_kms.cc:227] Using postgres at 127.27.187.148:44403
2026-08-06 09:05:59,820 [I] DB FLAVOR :POSTGRES
2026-08-06 09:05:59,821 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:05:59,821 [I] Checking connection
2026-08-06 09:05:59,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:06:00,125 [I] connection success
2026-08-06 09:06:00,125 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:06:00,125 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:06:00,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:06:00,400 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:06:00,401 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:06:00,401 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:06:00,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:44403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:06:00,691 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:06:01.280861 28398 mini_ranger_kms.cc:310] Created kms service
I20260806 09:06:01.516448 28398 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:06:01.573230 28398 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:06:01 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:02.108428 28398 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:06:02.110358 28398 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:06:02.110477 28398 mini_ranger_kms.cc:240] Using host: 127.27.187.148
I20260806 09:06:02.140785 28398 mini_ranger_kms.cc:276] Ranger KMS PID: 7485
I20260806 09:06:02.141021 28398 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.187.148:53433
Aug 06, 2026 9:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 09:06:04.653833 28398 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.610s user 0.147s sys 0.441s
I20260806 09:06:04.654237 28398 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:06:04.654464 28398 mini_ranger_kms.cc:401] 127.27.187.148:53433/kms/v1/keys
Aug 06 09:06:04 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 09:06:05 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
Aug 06 09:06:05 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
Aug 06 09:06:08 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:06:08.419265 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38795
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38795
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:53433/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:06:08.864280 7536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:06:08.864707 7536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:06:08.864995 7536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:06:08.874702 7536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:06:08.874979 7536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:06:08.875108 7536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:06:08.875286 7536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:06:08.875435 7536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:06:08.888473 7536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.187.148:53433/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38795
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:06:08.890208 7536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:06:08.892014 7536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:06:08.909631 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:08.909847 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:08.909685 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:06:08.910784 7536 server_base.cc:1034] running on GCE node
I20260806 09:06:08.911825 7536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:06:08.913679 7536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:06:08.914924 7536 hybrid_clock.cc:648] HybridClock initialized: now 1786007168914863 us; error 52 us; skew 500 ppm
I20260806 09:06:08.915413 7536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:06:08 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:06:08.919963 7536 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:06:08.922048 7536 webserver.cc:460] Webserver started at http://127.27.187.190:44727/ using document root <none> and password file <none>
I20260806 09:06:08.922971 7536 fs_manager.cc:359] Metadata directory not provided
I20260806 09:06:08.923244 7536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:06:08.923754 7536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:06:08 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:08.997359 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "25ddf690c3554660bee60c660772d71b"
format_stamp: "Formatted at 2026-08-06 09:06:08 on dist-test-slave-8tdl"
server_key: "c269830bc83f7d3cbdfce5f208125cb1"
server_key_iv: "92c0f57f47fc6c402af5f3af72de8be9"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:08.999243 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "25ddf690c3554660bee60c660772d71b"
format_stamp: "Formatted at 2026-08-06 09:06:08 on dist-test-slave-8tdl"
server_key: "c269830bc83f7d3cbdfce5f208125cb1"
server_key_iv: "92c0f57f47fc6c402af5f3af72de8be9"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:09.010540 7536 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20260806 09:06:09.034523 7551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:09.036671 7536 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:06:09.036984 7536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "25ddf690c3554660bee60c660772d71b"
format_stamp: "Formatted at 2026-08-06 09:06:08 on dist-test-slave-8tdl"
server_key: "c269830bc83f7d3cbdfce5f208125cb1"
server_key_iv: "92c0f57f47fc6c402af5f3af72de8be9"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:09.037338 7536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:06:09.081423 7536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:06:09.087160 7536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:06:09.087590 7536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:06:09.124902 7536 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38795
I20260806 09:06:09.124920 7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38795 every 8 connection(s)
I20260806 09:06:09.126811 7536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:06:09.131418 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 7536
I20260806 09:06:09.131693 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:06:09.132682 28398 external_mini_cluster.cc:1428] Setting key e843a921e215571697d6cfd82238769b
I20260806 09:06:09.132699 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:06:09.157384 7603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: Bootstrap starting.
I20260806 09:06:09.163200 7603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: Neither blocks nor log segments found. Creating new log.
Aug 06 09:06:09 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:06:09.165192 7603 log.cc:824] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:06:09.169309 7603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: No bootstrap required, opened a new log
I20260806 09:06:09.175273 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:09.135846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.27.187.190:38795)
0806 09:06:09.148178 (+ 12332us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:09.148200 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:09.148294 (+ 94us) server_negotiation.cc:407] Connection header received
0806 09:06:09.151316 (+ 3022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:09.151358 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:09.152077 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:09.152758 (+ 681us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:09.154127 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:09.156671 (+ 2544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:09.158263 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:09.160099 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:09.160386 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:09.165459 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:09.165474 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:09.165496 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:09.165601 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:09.169961 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:09.170984 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:09.170991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:09.170998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:09.171165 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:09.171663 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:09.171668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:09.171670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:09.172478 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:09.172789 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:09.173346 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 09:06:09.173842 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11780,"thread_start_us":14739,"threads_started":1}
I20260806 09:06:09.180526 7603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } }
I20260806 09:06:09.181036 7603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:06:09.181197 7603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ddf690c3554660bee60c660772d71b, State: Initialized, Role: FOLLOWER
I20260806 09:06:09.181958 7603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } }
I20260806 09:06:09.182358 7603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:06:09.182559 7603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:06:09.182835 7603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:06:09.185601 7603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } }
I20260806 09:06:09.186231 7603 leader_election.cc:302] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ddf690c3554660bee60c660772d71b; no voters:
I20260806 09:06:09.187481 7603 leader_election.cc:288] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:06:09.187750 7608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:06:09.188743 7608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [term 1 LEADER]: Becoming Leader. State: Replica: 25ddf690c3554660bee60c660772d71b, State: Running, Role: LEADER
I20260806 09:06:09.189298 7608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } }
I20260806 09:06:09.190006 7603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:06:09.192862 7609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ddf690c3554660bee60c660772d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } } }
I20260806 09:06:09.192898 7610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ddf690c3554660bee60c660772d71b. Latest consensus state: current_term: 1 leader_uuid: "25ddf690c3554660bee60c660772d71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ddf690c3554660bee60c660772d71b" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38795 } } }
I20260806 09:06:09.193574 7609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [sys.catalog]: This master's current role is: LEADER
I20260806 09:06:09.193583 7610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b [sys.catalog]: This master's current role is: LEADER
I20260806 09:06:09.198952 7613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:06:09.205681 7613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:06:09.206756 7603 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:06:09.226441 7613 catalog_manager.cc:1347] Generated new cluster ID: 727af84f19d24fe7b2e56bc9b09080bb
I20260806 09:06:09.226603 7613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:06:09.268687 7613 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:06:09.272099 7613 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:06:09.292418 7613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: Generated new TSK 0
I20260806 09:06:09.293691 7613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:06:10.200649 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:53433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:06:10 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:06:10.721823 7649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:06:10.722363 7649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:06:10.722568 7649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:06:10.736171 7649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:06:10.736425 7649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:06:10.736678 7649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:06:10.755643 7649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.187.148:53433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:06:10.757634 7649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:06:10.759610 7649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:06:10.779393 7659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:10.781075 7660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:10.789647 7662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:06:10.790838 7649 server_base.cc:1034] running on GCE node
I20260806 09:06:10.791913 7649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:06:10.793480 7649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:06:10.794806 7649 hybrid_clock.cc:648] HybridClock initialized: now 1786007170794737 us; error 61 us; skew 500 ppm
I20260806 09:06:10.795307 7649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:06:10 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:06:10.800429 7649 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:06:10.811818 7649 webserver.cc:460] Webserver started at http://127.27.187.129:40125/ using document root <none> and password file <none>
I20260806 09:06:10.813117 7649 fs_manager.cc:359] Metadata directory not provided
I20260806 09:06:10.813364 7649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:06:10.813832 7649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:06:10 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:10.845008 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "a4158043765246b7afa15244b67cb0b4"
format_stamp: "Formatted at 2026-08-06 09:06:10 on dist-test-slave-8tdl"
server_key: "a814e5d96fb6bb1ff07104998b3b82d6"
server_key_iv: "c39e15e924f3dd8a1f719a69956b14eb"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:10.846916 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "a4158043765246b7afa15244b67cb0b4"
format_stamp: "Formatted at 2026-08-06 09:06:10 on dist-test-slave-8tdl"
server_key: "a814e5d96fb6bb1ff07104998b3b82d6"
server_key_iv: "c39e15e924f3dd8a1f719a69956b14eb"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:10.855743 7649 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 09:06:10.875340 7669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:10.877305 7649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:06:10.877537 7649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "a4158043765246b7afa15244b67cb0b4"
format_stamp: "Formatted at 2026-08-06 09:06:10 on dist-test-slave-8tdl"
server_key: "a814e5d96fb6bb1ff07104998b3b82d6"
server_key_iv: "c39e15e924f3dd8a1f719a69956b14eb"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:10.877818 7649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:06:10.916013 7649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:06:10.923728 7649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:06:10.924258 7649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:06:10.925889 7649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:06:10.929885 7649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:06:10.930059 7649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:06:10.930332 7649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:06:10.930485 7649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:11.090934 7649 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43859
I20260806 09:06:11.091109 7782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43859 every 8 connection(s)
I20260806 09:06:11.093679 7649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:06:11.099181 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 7649
I20260806 09:06:11.099431 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:06:11.100451 28398 external_mini_cluster.cc:1428] Setting key 823ecff3459c9135da5b2eb3a111a8fc
Aug 06 09:06:11 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:06:11.176986 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:11.119554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:56809 (local address 127.27.187.190:38795)
0806 09:06:11.139300 (+ 19746us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:11.139311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:11.142787 (+ 3476us) server_negotiation.cc:407] Connection header received
0806 09:06:11.145594 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:11.145601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:11.145701 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:11.145813 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:11.149525 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:11.151324 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:11.155400 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:11.156386 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:11.156534 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:11.163632 (+ 7098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:11.163642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:11.163649 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:11.163755 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:11.170250 (+ 6495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:11.172336 (+ 2086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:11.172347 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:11.172351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:11.172627 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:11.173541 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:11.173547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:11.173551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:11.174183 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:11.174361 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:11.175800 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0806 09:06:11.176157 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19486,"thread_start_us":19768,"threads_started":1}
Aug 06 09:06:11 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:11.181802 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:11.118485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38795 (local address 127.27.187.129:56809)
0806 09:06:11.140809 (+ 22324us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:11.140873 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:11.144959 (+ 4086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:11.146472 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:11.146487 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:11.147595 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:06:11.149077 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:11.149196 (+ 119us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:11.151674 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:11.151707 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:11.155093 (+ 3386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:11.155121 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:11.157103 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:11.157116 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:11.157460 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:11.158310 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:06:11.158349 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:06:11.163120 (+ 4771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:06:11.170736 (+ 7616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:11.170750 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:11.170785 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:11.172066 (+ 1281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:11.172837 (+ 771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:11.172842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:11.172847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:11.173344 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:11.174523 (+ 1179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:06:11.174533 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:06:11.175200 (+ 667us) client_negotiation.cc:769] Sending connection context
0806 09:06:11.177853 (+ 2653us) client_negotiation.cc:240] Negotiation successful
0806 09:06:11.178756 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21860,"thread_start_us":21421,"threads_started":1}
I20260806 09:06:11.184845 7783 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38795
I20260806 09:06:11.185426 7783 heartbeater.cc:464] Registering TS with master...
I20260806 09:06:11.187160 7783 heartbeater.cc:511] Master 127.27.187.190:38795 requested a full tablet report, sending...
I20260806 09:06:11.190508 7568 ts_manager.cc:194] Registered new tserver with Master: a4158043765246b7afa15244b67cb0b4 (127.27.187.129:43859)
I20260806 09:06:11.193310 7568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:56809
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
W20260806 09:06:11.265684 28398 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.187.130: real 0.161s user 0.001s sys 0.025s
I20260806 09:06:11.267262 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:53433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:06:11 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
W20260806 09:06:12.115969 7789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:06:12.116463 7789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:06:12.116711 7789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:06:12.129664 7789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:06:12.129984 7789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:06:12.130327 7789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:06:12.147619 7789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.187.148:53433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:06:12.149473 7789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:06:12.151328 7789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:06:12.179598 7803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:12.180042 7804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:12.192860 7806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:06:12.193379 7789 server_base.cc:1034] running on GCE node
I20260806 09:06:12.194370 7789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:06:12.196044 7789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:06:12.197432 7789 hybrid_clock.cc:648] HybridClock initialized: now 1786007172197319 us; error 60 us; skew 500 ppm
I20260806 09:06:12.197974 7789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:06:12.198833 7783 heartbeater.cc:503] Master 127.27.187.190:38795 was elected leader, sending a full tablet report...
Aug 06 09:06:12 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:06:12.204406 7789 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:06:12.207932 7789 webserver.cc:460] Webserver started at http://127.27.187.130:41033/ using document root <none> and password file <none>
I20260806 09:06:12.209126 7789 fs_manager.cc:359] Metadata directory not provided
I20260806 09:06:12.209360 7789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:06:12.209915 7789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:06:12 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:12.234056 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d"
format_stamp: "Formatted at 2026-08-06 09:06:12 on dist-test-slave-8tdl"
server_key: "fa79277d4458c2ce6e8095d0565ccb48"
server_key_iv: "a1dcf86a33738ea1440c2079efd0dcfe"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:12.235726 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d"
format_stamp: "Formatted at 2026-08-06 09:06:12 on dist-test-slave-8tdl"
server_key: "fa79277d4458c2ce6e8095d0565ccb48"
server_key_iv: "a1dcf86a33738ea1440c2079efd0dcfe"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:12.267277 7789 fs_manager.cc:691] Time spent creating directory manager: real 0.031s user 0.011s sys 0.018s
I20260806 09:06:12.299621 7813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:12.302033 7789 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.003s
I20260806 09:06:12.302524 7789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d"
format_stamp: "Formatted at 2026-08-06 09:06:12 on dist-test-slave-8tdl"
server_key: "fa79277d4458c2ce6e8095d0565ccb48"
server_key_iv: "a1dcf86a33738ea1440c2079efd0dcfe"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:12.302965 7789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:06:12.328773 7789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:06:12.335093 7789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:06:12.335525 7789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:06:12.336928 7789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:06:12.339526 7789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:06:12.339689 7789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:12.339922 7789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:06:12.340056 7789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:12.509984 7789 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:44335
I20260806 09:06:12.510015 7926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:44335 every 8 connection(s)
I20260806 09:06:12.512535 7789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:06:12.519080 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 7789
I20260806 09:06:12.519280 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:06:12.519958 28398 external_mini_cluster.cc:1428] Setting key d0530d576e72e8e444aabffa7c76e162
Aug 06 09:06:12 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:06:12.548702 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:12.517134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:34105 (local address 127.27.187.190:38795)
0806 09:06:12.517880 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:12.517888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:12.519351 (+ 1463us) server_negotiation.cc:407] Connection header received
0806 09:06:12.522205 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:12.522210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:12.522291 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:12.522418 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:12.525088 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:12.526788 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:12.530325 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:12.531210 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:12.531364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:12.537256 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:12.537266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:12.537275 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:12.537356 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:12.543327 (+ 5971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:12.544720 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:12.544730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:12.544736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:12.545070 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:12.545824 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:12.545832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:12.545838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:12.546434 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:12.546629 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:12.547671 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0806 09:06:12.548003 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":286,"threads_started":1}
I20260806 09:06:12.549444 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:12.517631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38795 (local address 127.27.187.130:34105)
0806 09:06:12.519167 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:12.519216 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:12.521826 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:12.522731 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:12.522744 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:12.523628 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:06:12.524835 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:12.524867 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:12.527045 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:12.527054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:12.530077 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:12.530125 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:12.531564 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:12.531575 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:12.532029 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:12.533071 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:06:12.533134 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:06:12.536873 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:06:12.543701 (+ 6828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:12.543721 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:12.543740 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:12.544497 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:12.545262 (+ 765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:12.545266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:12.545269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:12.545651 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:12.546633 (+ 982us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:06:12.546641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:06:12.547158 (+ 517us) client_negotiation.cc:769] Sending connection context
0806 09:06:12.547451 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 09:06:12.547969 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":725,"threads_started":1}
I20260806 09:06:12.551676 7927 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38795
I20260806 09:06:12.552193 7927 heartbeater.cc:464] Registering TS with master...
I20260806 09:06:12.553673 7927 heartbeater.cc:511] Master 127.27.187.190:38795 requested a full tablet report, sending...
I20260806 09:06:12.556118 7568 ts_manager.cc:194] Registered new tserver with Master: 1cc6710fe9e94c1d8f9e01e53bb1e80d (127.27.187.130:44335)
I20260806 09:06:12.557128 7568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:34105
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:06:12.600147 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:53433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:06:13.207989 7933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:06:13.208519 7933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:06:13.208865 7933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:06:13.219703 7933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:06:13.219973 7933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:06:13.220218 7933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:06:13.233888 7933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.187.148:53433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:06:13.235872 7933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:06:13.237784 7933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:06:13.257861 7941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:13.258108 7938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:06:13.258085 7939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:06:13.260465 7933 server_base.cc:1034] running on GCE node
I20260806 09:06:13.261283 7933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:06:13.262667 7933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:06:13.264147 7933 hybrid_clock.cc:648] HybridClock initialized: now 1786007173263991 us; error 107 us; skew 500 ppm
I20260806 09:06:13.264593 7933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:06:13 dist-test-slave-8tdl krb5kdc[7086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:06:13.269325 7933 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:06:13.271379 7933 webserver.cc:460] Webserver started at http://127.27.187.131:37371/ using document root <none> and password file <none>
I20260806 09:06:13.272426 7933 fs_manager.cc:359] Metadata directory not provided
I20260806 09:06:13.272624 7933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:06:13.272987 7933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:06:13 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:13.292258 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "f5e820c00718410d97dca4df85b9e152"
format_stamp: "Formatted at 2026-08-06 09:06:13 on dist-test-slave-8tdl"
server_key: "0e6da05167c82755cd1885790e576072"
server_key_iv: "025442f73b0b4a35044a58a6b7e186d0"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:13.293496 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "f5e820c00718410d97dca4df85b9e152"
format_stamp: "Formatted at 2026-08-06 09:06:13 on dist-test-slave-8tdl"
server_key: "0e6da05167c82755cd1885790e576072"
server_key_iv: "025442f73b0b4a35044a58a6b7e186d0"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:13.301546 7933 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 09:06:13.320045 7948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:13.322049 7933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:06:13.322336 7933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "f5e820c00718410d97dca4df85b9e152"
format_stamp: "Formatted at 2026-08-06 09:06:13 on dist-test-slave-8tdl"
server_key: "0e6da05167c82755cd1885790e576072"
server_key_iv: "025442f73b0b4a35044a58a6b7e186d0"
server_key_version: "kuduclusterkey@0"
I20260806 09:06:13.322611 7933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:06:13.347436 7933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:06:13.353880 7933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:06:13.354389 7933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:06:13.355695 7933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:06:13.357975 7933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:06:13.358156 7933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:13.358530 7933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:06:13.358666 7933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:06:13.440405 7933 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:43771
I20260806 09:06:13.440513 8060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:43771 every 8 connection(s)
I20260806 09:06:13.441977 7933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:06:13.450912 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 7933
I20260806 09:06:13.451087 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:06:13.451681 28398 external_mini_cluster.cc:1428] Setting key 24478a7b4de20d7fe732af53247d4a58
Aug 06 09:06:13 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:06:13.475464 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.446363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:37115 (local address 127.27.187.190:38795)
0806 09:06:13.447297 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.447303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.449519 (+ 2216us) server_negotiation.cc:407] Connection header received
0806 09:06:13.452904 (+ 3385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.452910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.453006 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.453174 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:13.455921 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.457228 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.459706 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.460395 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.460515 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.465589 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:13.465596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:13.465601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:13.465669 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:13.471261 (+ 5592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:13.472224 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:13.472232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:13.472238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:13.472555 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:13.473132 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:13.473136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:13.473138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:13.473559 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:13.473706 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.474823 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.475037 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":608,"threads_started":1}
I20260806 09:06:13.476447 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.447456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38795 (local address 127.27.187.131:37115)
0806 09:06:13.449336 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.449376 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.452511 (+ 3135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.453595 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.453605 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.454376 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:06:13.455690 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.455722 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.457458 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.457463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.459455 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.459495 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.460801 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.460810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.461247 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.462188 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:06:13.462238 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:06:13.465176 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:06:13.471461 (+ 6285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:13.471470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:13.471488 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:13.472027 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:13.472737 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:06:13.472742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:06:13.472745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:06:13.472994 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:06:13.473781 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:06:13.473788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:06:13.474234 (+ 446us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.474531 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.475094 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":1142,"threads_started":1}
I20260806 09:06:13.478500 8061 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38795
I20260806 09:06:13.478976 8061 heartbeater.cc:464] Registering TS with master...
I20260806 09:06:13.480010 8061 heartbeater.cc:511] Master 127.27.187.190:38795 requested a full tablet report, sending...
I20260806 09:06:13.482774 7567 ts_manager.cc:194] Registered new tserver with Master: f5e820c00718410d97dca4df85b9e152 (127.27.187.131:43771)
I20260806 09:06:13.483824 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:06:13.484509 7567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:37115
I20260806 09:06:13.509932 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.491131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.27.187.190:38795)
0806 09:06:13.491514 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.491523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.492155 (+ 632us) server_negotiation.cc:407] Connection header received
0806 09:06:13.492490 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.492494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.492571 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.492688 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:13.494893 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.495727 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.497503 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.498223 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.498387 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.502066 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:13.502072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:13.502076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:13.502115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:13.505748 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:13.506806 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:13.506812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:13.506814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:13.507005 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:13.507504 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:13.507508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:13.507510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:13.507874 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:13.508245 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.508831 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.509329 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 09:06:13.515815 7567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:06:13.526535 7567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:06:13.560830 7927 heartbeater.cc:503] Master 127.27.187.190:38795 was elected leader, sending a full tablet report...
I20260806 09:06:13.567137 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.545115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:43771 (local address 127.0.0.1:52768)
0806 09:06:13.546886 (+ 1771us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.546927 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.547193 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.549082 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.549091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.549127 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.549903 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.549915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.555882 (+ 5967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.555892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.560328 (+ 4436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.560357 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.564667 (+ 4310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.564677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.565583 (+ 906us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.565703 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.565957 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.566257 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":1,"thread_start_us":1279,"threads_started":1}
I20260806 09:06:13.567744 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.545614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.27.187.131:43771)
0806 09:06:13.548337 (+ 2723us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.548348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.548406 (+ 58us) server_negotiation.cc:407] Connection header received
0806 09:06:13.548555 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.548566 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.548794 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.549100 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.550305 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.555590 (+ 5285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.560718 (+ 5128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.563189 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.565187 (+ 1998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.565357 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.566807 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.567063 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2481,"thread_start_us":2741,"threads_started":1}
I20260806 09:06:13.571455 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.545534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:44335 (local address 127.0.0.1:56992)
0806 09:06:13.551105 (+ 5571us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.551130 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.551555 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.552960 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.552969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.553011 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.554089 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.554110 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.560032 (+ 5922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.560042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.566155 (+ 6113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.566185 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.569779 (+ 3594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.569787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.570162 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.570225 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.570388 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.570601 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5330,"thread_start_us":4892,"threads_started":1}
I20260806 09:06:13.571715 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.546410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.27.187.130:44335)
0806 09:06:13.548311 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.548322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.551279 (+ 2957us) server_negotiation.cc:407] Connection header received
0806 09:06:13.551885 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.551896 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.552331 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.552664 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.554429 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.559709 (+ 5280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.566470 (+ 6761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.569237 (+ 2767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.569609 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.569848 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.570450 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.570743 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":1394,"threads_started":1}
I20260806 09:06:13.573840 7996 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 (DEFAULT_TABLE table=test-table [id=eec15f1602b74b97814037cb9730681b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:06:13.575587 7996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1f0872fbed42fb80c2a66eb4485399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:06:13.577791 7861 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 (DEFAULT_TABLE table=test-table [id=eec15f1602b74b97814037cb9730681b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:06:13.580509 7861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1f0872fbed42fb80c2a66eb4485399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:06:13.581480 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.551703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.27.187.129:43859)
0806 09:06:13.564480 (+ 12777us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.564491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.566235 (+ 1744us) server_negotiation.cc:407] Connection header received
0806 09:06:13.566784 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.566794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.567242 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.567482 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.569279 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.572743 (+ 3464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.576803 (+ 4060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.579747 (+ 2944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.580040 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.580286 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.580786 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.581043 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12483,"thread_start_us":12205,"threads_started":1}
I20260806 09:06:13.581622 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.550961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43859 (local address 127.0.0.1:59032)
0806 09:06:13.566092 (+ 15131us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.566118 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.566401 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.568011 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.568016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.568047 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.568917 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.568933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.573012 (+ 4079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.573017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.576547 (+ 3530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.576570 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.580152 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.580173 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.580511 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.580550 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.580661 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.580832 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14867,"thread_start_us":14504,"threads_started":1}
I20260806 09:06:13.587054 7717 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 (DEFAULT_TABLE table=test-table [id=eec15f1602b74b97814037cb9730681b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:06:13.589286 7717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1f0872fbed42fb80c2a66eb4485399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:06:13.599553 8080 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Bootstrap starting.
I20260806 09:06:13.599553 8081 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Bootstrap starting.
I20260806 09:06:13.604851 8080 tablet_bootstrap.cc:654] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Neither blocks nor log segments found. Creating new log.
I20260806 09:06:13.606109 8081 tablet_bootstrap.cc:654] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Neither blocks nor log segments found. Creating new log.
I20260806 09:06:13.606841 8080 log.cc:824] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Log is configured to *not* fsync() on all Append() calls
I20260806 09:06:13.608541 8081 log.cc:824] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:06:13.612612 8080 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: No bootstrap required, opened a new log
I20260806 09:06:13.612690 8085 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Bootstrap starting.
I20260806 09:06:13.614487 8080 ts_tablet_manager.cc:1397] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260806 09:06:13.614697 8081 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: No bootstrap required, opened a new log
I20260806 09:06:13.615144 8081 ts_tablet_manager.cc:1397] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260806 09:06:13.617290 8085 tablet_bootstrap.cc:654] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Neither blocks nor log segments found. Creating new log.
I20260806 09:06:13.618880 8085 log.cc:824] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:06:13.623088 8085 tablet_bootstrap.cc:492] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: No bootstrap required, opened a new log
I20260806 09:06:13.623535 8085 ts_tablet_manager.cc:1397] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 09:06:13.633524 8080 raft_consensus.cc:348] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.633514 8081 raft_consensus.cc:348] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.634089 8081 raft_consensus.cc:374] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:06:13.634110 8080 raft_consensus.cc:374] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:06:13.633430 8085 raft_consensus.cc:348] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.634317 8085 raft_consensus.cc:374] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:06:13.634308 8080 raft_consensus.cc:729] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e820c00718410d97dca4df85b9e152, State: Initialized, Role: FOLLOWER
I20260806 09:06:13.634308 8081 raft_consensus.cc:729] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc6710fe9e94c1d8f9e01e53bb1e80d, State: Initialized, Role: FOLLOWER
I20260806 09:06:13.634438 8085 raft_consensus.cc:729] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4158043765246b7afa15244b67cb0b4, State: Initialized, Role: FOLLOWER
I20260806 09:06:13.635197 8085 consensus_queue.cc:260] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.635227 8081 consensus_queue.cc:260] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.635248 8080 consensus_queue.cc:260] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.637395 8085 ts_tablet_manager.cc:1428] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 09:06:13.638593 8061 heartbeater.cc:503] Master 127.27.187.190:38795 was elected leader, sending a full tablet report...
I20260806 09:06:13.639559 8080 ts_tablet_manager.cc:1428] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Time spent starting tablet: real 0.025s user 0.019s sys 0.005s
I20260806 09:06:13.639874 8081 ts_tablet_manager.cc:1428] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Time spent starting tablet: real 0.024s user 0.016s sys 0.007s
W20260806 09:06:13.695407 8062 tablet.cc:2369] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:06:13.766618 7928 tablet.cc:2369] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:06:13.835940 8089 raft_consensus.cc:482] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:06:13.836325 8089 raft_consensus.cc:504] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.838282 8089 leader_election.cc:288] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e820c00718410d97dca4df85b9e152 (127.27.187.131:43771), a4158043765246b7afa15244b67cb0b4 (127.27.187.129:43859)
I20260806 09:06:13.846400 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.838942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:52803 (local address 127.27.187.131:43771)
0806 09:06:13.839187 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.839196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.839922 (+ 726us) server_negotiation.cc:407] Connection header received
0806 09:06:13.840130 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.840139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.840243 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.840322 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.840988 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.841942 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.844375 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.845325 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.845521 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.845596 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.845971 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.846116 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:06:13.846546 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.838536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:43771 (local address 127.27.187.130:52803)
0806 09:06:13.839775 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.839792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.839932 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.840359 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.840364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.840389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.840810 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.840821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.842227 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.842234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.844199 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.844213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.845571 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.845577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.845776 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.845817 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.845932 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.846086 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":5,"thread_start_us":948,"threads_started":1}
I20260806 09:06:13.847048 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.838746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43859 (local address 127.27.187.130:35791)
0806 09:06:13.839871 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.839895 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.840079 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.840545 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.840549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.840581 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.841105 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.841122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.842476 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.842480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.844197 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.844219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.845844 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.845852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.846056 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.846095 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.846185 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.846330 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":48,"thread_start_us":664,"threads_started":1}
I20260806 09:06:13.847682 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e820c00718410d97dca4df85b9e152" is_pre_election: true
I20260806 09:06:13.848044 8088 raft_consensus.cc:482] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:06:13.848143 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.838935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:35791 (local address 127.27.187.129:43859)
0806 09:06:13.839193 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.839203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.840030 (+ 827us) server_negotiation.cc:407] Connection header received
0806 09:06:13.840296 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.840305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.840389 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.840495 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.841301 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.842289 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.844445 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.845453 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.845631 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.845711 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.846877 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.847040 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:06:13.848444 8088 raft_consensus.cc:504] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.849267 8016 raft_consensus.cc:2454] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in term 0.
I20260806 09:06:13.849397 7737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4158043765246b7afa15244b67cb0b4" is_pre_election: true
I20260806 09:06:13.850014 7737 raft_consensus.cc:2454] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in term 0.
I20260806 09:06:13.850476 8088 leader_election.cc:288] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc6710fe9e94c1d8f9e01e53bb1e80d (127.27.187.130:44335), a4158043765246b7afa15244b67cb0b4 (127.27.187.129:43859)
I20260806 09:06:13.851277 7815 leader_election.cc:302] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc6710fe9e94c1d8f9e01e53bb1e80d, f5e820c00718410d97dca4df85b9e152; no voters:
I20260806 09:06:13.851800 8089 raft_consensus.cc:2781] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:06:13.851956 8089 raft_consensus.cc:482] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:06:13.852090 8089 raft_consensus.cc:3037] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:06:13.855651 8089 raft_consensus.cc:504] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.856902 8089 leader_election.cc:288] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [CANDIDATE]: Term 1 election: Requested vote from peers f5e820c00718410d97dca4df85b9e152 (127.27.187.131:43771), a4158043765246b7afa15244b67cb0b4 (127.27.187.129:43859)
I20260806 09:06:13.857244 7737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4158043765246b7afa15244b67cb0b4"
I20260806 09:06:13.857278 8016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e820c00718410d97dca4df85b9e152"
I20260806 09:06:13.857506 8016 raft_consensus.cc:3037] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:06:13.857511 7737 raft_consensus.cc:3037] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:06:13.858852 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.850889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:44335 (local address 127.27.187.131:49095)
0806 09:06:13.851134 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.851155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.851276 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.852185 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.852193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.852230 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.852458 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.852473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.853734 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.853739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.856165 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.856192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.858013 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.858017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.858188 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.858237 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.858351 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.858502 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1}
I20260806 09:06:13.858943 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.851517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49095 (local address 127.27.187.130:44335)
0806 09:06:13.851722 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.851729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.851748 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:06:13.851825 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.851830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.851935 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.852062 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.852622 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.853582 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.856389 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.857570 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.857751 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.857805 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.858380 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.858530 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:06:13.860569 7881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "f5e820c00718410d97dca4df85b9e152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" is_pre_election: true
I20260806 09:06:13.861472 7881 raft_consensus.cc:2379] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5e820c00718410d97dca4df85b9e152 in current term 1: Already voted for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in this term.
I20260806 09:06:13.862823 7737 raft_consensus.cc:2454] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in term 1.
I20260806 09:06:13.863183 8016 raft_consensus.cc:2454] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in term 1.
I20260806 09:06:13.863498 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.851599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:33303 (local address 127.27.187.129:43859)
0806 09:06:13.851791 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.851800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.852249 (+ 449us) server_negotiation.cc:407] Connection header received
0806 09:06:13.852512 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.852517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.852593 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.852683 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:06:13.853443 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.854270 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.856426 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.860631 (+ 4205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.860938 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.861144 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.862799 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.862992 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:06:13.863497 7817 leader_election.cc:302] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc6710fe9e94c1d8f9e01e53bb1e80d, a4158043765246b7afa15244b67cb0b4; no voters:
I20260806 09:06:13.863202 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.851297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43859 (local address 127.27.187.131:33303)
0806 09:06:13.852145 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0806 09:06:13.852164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:06:13.852338 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:06:13.852851 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:06:13.852856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:06:13.852884 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:06:13.853241 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.853255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.854452 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.854456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.856253 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:06:13.856288 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.861447 (+ 5159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.861454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:06:13.861804 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.861861 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 09:06:13.861983 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 09:06:13.862148 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":1,"thread_start_us":529,"threads_started":1}
I20260806 09:06:13.864125 8089 raft_consensus.cc:2781] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:06:13.864632 8089 raft_consensus.cc:686] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 LEADER]: Becoming Leader. State: Replica: 1cc6710fe9e94c1d8f9e01e53bb1e80d, State: Running, Role: LEADER
I20260806 09:06:13.864671 7737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1f0872fbed42fb80c2a66eb4485399" candidate_uuid: "f5e820c00718410d97dca4df85b9e152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4158043765246b7afa15244b67cb0b4" is_pre_election: true
I20260806 09:06:13.865096 7737 raft_consensus.cc:2379] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5e820c00718410d97dca4df85b9e152 in current term 1: Already voted for candidate 1cc6710fe9e94c1d8f9e01e53bb1e80d in this term.
I20260806 09:06:13.865350 8089 consensus_queue.cc:237] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } }
I20260806 09:06:13.865844 7952 leader_election.cc:302] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5e820c00718410d97dca4df85b9e152; no voters: 1cc6710fe9e94c1d8f9e01e53bb1e80d, a4158043765246b7afa15244b67cb0b4
I20260806 09:06:13.866389 8088 raft_consensus.cc:2726] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:06:13.870905 7567 catalog_manager.cc:5495] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cc6710fe9e94c1d8f9e01e53bb1e80d (127.27.187.130). New cstate: current_term: 1 leader_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc6710fe9e94c1d8f9e01e53bb1e80d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 } health_report { overall_health: UNKNOWN } } }
W20260806 09:06:13.879598 7784 tablet.cc:2369] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:06:13.935309 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:13.928137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.27.187.130:44335)
0806 09:06:13.928341 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:13.928347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:13.928367 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:06:13.928490 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:13.928495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:13.928580 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:13.928729 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:06:13.929383 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.930376 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.932391 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:13.933092 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:13.933229 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:13.933866 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:06:13.934542 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:06:13.934668 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:13.934873 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 09:06:13.934957 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:06:13.946395 7737 raft_consensus.cc:1261] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 1 FOLLOWER]: Refusing update from remote peer 1cc6710fe9e94c1d8f9e01e53bb1e80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:06:13.946609 8016 raft_consensus.cc:1261] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 1 FOLLOWER]: Refusing update from remote peer 1cc6710fe9e94c1d8f9e01e53bb1e80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:06:13.947695 8093 consensus_queue.cc:1035] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4158043765246b7afa15244b67cb0b4" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:06:13.948278 8089 consensus_queue.cc:1035] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e820c00718410d97dca4df85b9e152" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:06:13.964217 8096 mvcc.cc:205] Tried to move back new op lower bound from 7315485384472305664 to 7315485384157057024. Current Snapshot: MvccSnapshot[applied={T|T < 7315485384472305664}]
I20260806 09:06:13.987090 7567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:06:13.987517 7567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:06:13.993696 7567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25ddf690c3554660bee60c660772d71b: Sending DeleteTablet for 3 replicas of tablet 6d1f0872fbed42fb80c2a66eb4485399
I20260806 09:06:13.995543 7996 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:06:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:52768
I20260806 09:06:13.995852 7861 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:06:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:56992
I20260806 09:06:13.995843 7717 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1f0872fbed42fb80c2a66eb4485399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:06:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:59032
I20260806 09:06:13.997802 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 7649
I20260806 09:06:14.012408 8106 tablet_replica.cc:330] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: stopping tablet replica
I20260806 09:06:14.013060 8106 raft_consensus.cc:2229] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:06:14.013464 8107 tablet_replica.cc:330] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: stopping tablet replica
I20260806 09:06:14.014073 8106 raft_consensus.cc:2258] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:06:14.014607 8107 raft_consensus.cc:2229] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:06:14.015818 8107 raft_consensus.cc:2258] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:06:14.017802 8106 ts_tablet_manager.cc:1905] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:06:14.018338 8107 ts_tablet_manager.cc:1905] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:06:14.018424 8108 tablet_replica.cc:330] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: stopping tablet replica
I20260806 09:06:14.018934 8108 raft_consensus.cc:2229] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:06:14.027542 8108 raft_consensus.cc:2258] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 09:06:14 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:06:14.030471 8108 ts_tablet_manager.cc:1905] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:06:14.030447 8107 ts_tablet_manager.cc:1918] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:06:14.030870 8107 log.cc:1196] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/6d1f0872fbed42fb80c2a66eb4485399
I20260806 09:06:14.032399 8106 ts_tablet_manager.cc:1918] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:06:14.032682 8106 log.cc:1196] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/6d1f0872fbed42fb80c2a66eb4485399
I20260806 09:06:14.033249 8107 ts_tablet_manager.cc:1939] T 6d1f0872fbed42fb80c2a66eb4485399 P 1cc6710fe9e94c1d8f9e01e53bb1e80d: Deleting consensus metadata
I20260806 09:06:14.033531 8106 ts_tablet_manager.cc:1939] T 6d1f0872fbed42fb80c2a66eb4485399 P f5e820c00718410d97dca4df85b9e152: Deleting consensus metadata
I20260806 09:06:14.036012 7553 catalog_manager.cc:4841] TS f5e820c00718410d97dca4df85b9e152 (127.27.187.131:43771): tablet 6d1f0872fbed42fb80c2a66eb4485399 (table test-table [id=eec15f1602b74b97814037cb9730681b]) successfully deleted
I20260806 09:06:14.038842 7555 catalog_manager.cc:4841] TS 1cc6710fe9e94c1d8f9e01e53bb1e80d (127.27.187.130:44335): tablet 6d1f0872fbed42fb80c2a66eb4485399 (table test-table [id=eec15f1602b74b97814037cb9730681b]) successfully deleted
I20260806 09:06:14.039796 8108 ts_tablet_manager.cc:1918] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:06:14.040002 8108 log.cc:1196] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/6d1f0872fbed42fb80c2a66eb4485399
I20260806 09:06:14.041214 8108 ts_tablet_manager.cc:1939] T 6d1f0872fbed42fb80c2a66eb4485399 P a4158043765246b7afa15244b67cb0b4: Deleting consensus metadata
I20260806 09:06:14.043654 7555 catalog_manager.cc:4841] TS a4158043765246b7afa15244b67cb0b4 (127.27.187.129:43859): tablet 6d1f0872fbed42fb80c2a66eb4485399 (table test-table [id=eec15f1602b74b97814037cb9730681b]) successfully deleted
I20260806 09:06:14.044533 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:14.004900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.27.187.129:43859)
0806 09:06:14.005196 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:14.005207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:14.012204 (+ 6997us) server_negotiation.cc:407] Connection header received
0806 09:06:14.012796 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:14.012807 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:14.012954 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:14.013067 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:14.016372 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.017715 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.020082 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.020941 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.021130 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:14.030042 (+ 8912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:14.030062 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:14.030083 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:14.030190 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:14.039025 (+ 8835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.040710 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.040723 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.040727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.041010 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.041733 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.041739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.041743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.042451 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:14.043020 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:14.043763 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 09:06:14.044059 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 09:06:14.045722 7777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:06:14.222023 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 7649
I20260806 09:06:14.246670 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 7789
Aug 06 09:06:14 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:06:14.267463 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:14.247657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.27.187.130:44335)
0806 09:06:14.248162 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:14.248172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:14.248315 (+ 143us) server_negotiation.cc:407] Connection header received
0806 09:06:14.248843 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:14.248848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:14.248949 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:14.249048 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:14.251187 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.251993 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.254199 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.254728 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.254859 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:14.260734 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:14.260751 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:14.260768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:14.260816 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:14.264237 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.265061 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.265070 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.265072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.265239 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.265697 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.265712 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.265714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.266113 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:14.266488 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:14.266853 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 09:06:14.267116 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
I20260806 09:06:14.268287 7922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:06:14.388041 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 7789
I20260806 09:06:14.413218 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 7933
Aug 06 09:06:14 dist-test-slave-8tdl krb5kdc[7086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:06:14.432497 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:06:14.414172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.27.187.131:43771)
0806 09:06:14.414768 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0806 09:06:14.414774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:06:14.414796 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:06:14.414855 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:06:14.414859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:06:14.414926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:06:14.415030 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:06:14.416763 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.417601 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.419385 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:06:14.419868 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:06:14.419983 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:06:14.425318 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:06:14.425338 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:06:14.425356 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:06:14.425425 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:06:14.429432 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.430334 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.430342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.430345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.430514 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:06:14.431003 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:06:14.431007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:06:14.431009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:06:14.431416 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:06:14.431701 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:06:14.431962 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 09:06:14.432215 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":292,"threads_started":1}
I20260806 09:06:14.433252 8056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:06:14.528985 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 7933
I20260806 09:06:14.550084 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 7536
I20260806 09:06:14.551407 7598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:06:14.682889 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 7536
I20260806 09:06:14.710016 28398 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:06:15.057107 28398 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:06:15.057442 28398 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:06:15.404788 28398 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:06:15.405 UTC [7116] LOG: received smart shutdown request
2026-08-06 09:06:15.410 UTC [7116] LOG: background worker "logical replication launcher" (PID 7125) exited with exit code 1
2026-08-06 09:06:15.410 UTC [7120] LOG: shutting down
2026-08-06 09:06:15.647 UTC [7116] LOG: database system is shut down
2026-08-06T09:06:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58672 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:06:15 dist-test-slave-8tdl krb5kdc[8123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:06:15 dist-test-slave-8tdl krb5kdc[8123](info): set up 2 sockets
Aug 06 09:06:15 dist-test-slave-8tdl krb5kdc[8123](info): commencing operation
krb5kdc: starting...
W20260806 09:06:16.969755 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:06:17 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:06:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangeradmin_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangeradmin_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangerlookup_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangerlookup_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
I20260806 09:06:17.407016 28398 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvYHTS_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/postgres -l logfile start
2026-08-06 09:06:20.429 UTC [8153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:06:20.429 UTC [8153] LOG: listening on IPv4 address "127.27.187.148", port 52065
2026-08-06 09:06:20.435 UTC [8153] LOG: listening on Unix socket "/tmp/.s.PGSQL.52065"
2026-08-06 09:06:20.458 UTC [8156] LOG: database system was shut down at 2026-08-06 09:06:18 UTC
2026-08-06 09:06:20.463 UTC [8153] LOG: database system is ready to accept connections
I20260806 09:06:21.501531 28398 mini_postgres.cc:96] Postgres bound to 52065
2026-08-06 09:06:21.526 UTC [8164] FATAL: database "slave" does not exist
127.27.187.148:52065 - accepting connections
I20260806 09:06:21.527062 28398 mini_ranger.cc:161] Starting Ranger...
I20260806 09:06:21.562218 28398 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:06:22.236481 28398 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:06:22.236619 28398 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin
2026-08-06 09:06:22,535 [I] DB FLAVOR :POSTGRES
2026-08-06 09:06:22,535 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:06:22,535 [I] Checking connection..
2026-08-06 09:06:22,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:06:22,790 [I] Checking connection passed.
2026-08-06 09:06:22,790 [I] --------- Verifying version history table ---------
2026-08-06 09:06:22,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:06:23,070 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:06:23,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:06:23,334 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:06:23,334 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:06:23,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:06:23.587 UTC [8229] WARNING: there is no transaction in progress
2026-08-06 09:06:23,595 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:06:23,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:06:23,882 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:06:23,882 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:06:23,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:06:24,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:06:24,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-06 09:06:24,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:06:24,968 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:06:24,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:06:25,258 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:06:25,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:06:25,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:06:25,801 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:06:25,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:06:26.917 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:26.938 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:26.958 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:26.976 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:26.994 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.019 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.080 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.089 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.106 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.120 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.132 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.148 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.160 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.169 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.178 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:27.187 UTC [8337] WARNING: there is no transaction in progress
2026-08-06 09:06:28,041 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:06:28,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:06:28,292 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:06:28,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:06:28,560 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:06:28,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:06:28,823 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:06:28,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:06:29,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:06:29,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-06 09:06:29,619 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:06:29,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:06:29,894 [I] DB_PATCHES have already been applied
I20260806 09:06:29.901060 28398 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvYHTS_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:06:29.901165 28398 mini_ranger.cc:193] Using host: 127.27.187.148
I20260806 09:06:29.922830 28398 mini_ranger.cc:227] Ranger admin URL: http://127.27.187.148:58063
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:06:31.133147 28398 mini_ranger.cc:160] Time spent starting Ranger: real 9.606s user 0.000s sys 0.065s
Aug 06, 2026 9:06:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:06:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 9:06:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:06:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:06:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:06:54.136711 28398 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.187.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.187.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangerkms_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.187.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/rangerkms_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.187.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.187.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.187.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/HTTP_127.27.187.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:06:54 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:06:54.361176 28398 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:06:54.389882 28398 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:06:55.488157 28398 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:06:55.488389 28398 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms
I20260806 09:06:55.488451 28398 mini_ranger_kms.cc:227] Using postgres at 127.27.187.148:52065
2026-08-06 09:06:55,878 [I] DB FLAVOR :POSTGRES
2026-08-06 09:06:55,878 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:06:55,878 [I] Checking connection
2026-08-06 09:06:55,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:06:56,150 [I] connection success
2026-08-06 09:06:56,150 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:06:56,150 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:06:56,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:06:56,428 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:06:56,428 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:06:56,428 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:06:56,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.187.148:52065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:06:56,760 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:06:57.334221 28398 mini_ranger_kms.cc:310] Created kms service
I20260806 09:06:57.513340 28398 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:06:57.569963 28398 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:06:57 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:06:58.044606 28398 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:06:58.044960 28398 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvYHTS_/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvYHTS_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:06:58.045011 28398 mini_ranger_kms.cc:240] Using host: 127.27.187.148
I20260806 09:06:58.071781 28398 mini_ranger_kms.cc:276] Ranger KMS PID: 8522
I20260806 09:06:58.071961 28398 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.187.148:58257
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:06:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 09:07:00 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:00.508558 28398 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.147s user 0.100s sys 0.327s
I20260806 09:07:00.508857 28398 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:07:00.509045 28398 mini_ranger_kms.cc:401] 127.27.187.148:58257/kms/v1/keys
Aug 06 09:07:00 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
Aug 06 09:07:01 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007220, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.187.148@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
Aug 06 09:07:03 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:07:03.968613 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45283
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45283
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:04.341796 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:04.342180 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:04.342337 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:04.350184 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:04.350354 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:04.350449 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:04.350549 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:04.350637 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:04.361690 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.187.148:58257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45283
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:04.363229 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:04.364663 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:04.379076 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:04.379101 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:04.379823 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:04.379786 8572 server_base.cc:1034] running on GCE node
I20260806 09:07:04.380803 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:04.382452 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:04.383730 8572 hybrid_clock.cc:648] HybridClock initialized: now 1786007224383678 us; error 44 us; skew 500 ppm
I20260806 09:07:04.384122 8572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:04 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:04.401714 8572 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:04.403720 8572 webserver.cc:460] Webserver started at http://127.27.187.190:36787/ using document root <none> and password file <none>
I20260806 09:07:04.404472 8572 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:04.404603 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:04.404954 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:07:04 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:04.473912 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "9bd351b4e2fa46339d8a0b55d6db324d"
format_stamp: "Formatted at 2026-08-06 09:07:04 on dist-test-slave-8tdl"
server_key: "37c99f43776fe12bb8d8d36a8c5e9599"
server_key_iv: "a9b87a2054e3a1d7bd42c4936f7969f8"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:04.474998 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "9bd351b4e2fa46339d8a0b55d6db324d"
format_stamp: "Formatted at 2026-08-06 09:07:04 on dist-test-slave-8tdl"
server_key: "37c99f43776fe12bb8d8d36a8c5e9599"
server_key_iv: "a9b87a2054e3a1d7bd42c4936f7969f8"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:04.481175 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:07:04.501389 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:04.503453 8572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:07:04.503686 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "9bd351b4e2fa46339d8a0b55d6db324d"
format_stamp: "Formatted at 2026-08-06 09:07:04 on dist-test-slave-8tdl"
server_key: "37c99f43776fe12bb8d8d36a8c5e9599"
server_key_iv: "a9b87a2054e3a1d7bd42c4936f7969f8"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:04.503984 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:04.539968 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:04.545722 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:04.546105 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:04.569291 8572 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45283
I20260806 09:07:04.569317 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45283 every 8 connection(s)
I20260806 09:07:04.571405 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:04.574180 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 8572
I20260806 09:07:04.574438 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:07:04.575381 28398 external_mini_cluster.cc:1428] Setting key 1de3b5695d45cb0192f2f940a674bfb3
I20260806 09:07:04.575990 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:04.594571 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Bootstrap starting.
I20260806 09:07:04.598654 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:04.599884 8639 log.cc:824] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:04.606110 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: No bootstrap required, opened a new log
Aug 06 09:07:04 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:04.616992 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:04.617646 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:04.617899 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd351b4e2fa46339d8a0b55d6db324d, State: Initialized, Role: FOLLOWER
I20260806 09:07:04.618882 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:04.619174 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:04.619390 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:04.619573 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:04.620002 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:04.577616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.27.187.190:45283)
0806 09:07:04.589569 (+ 11953us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:04.589588 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:04.591468 (+ 1880us) server_negotiation.cc:407] Connection header received
0806 09:07:04.594497 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:04.594535 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:04.595267 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:04.595812 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:04.597729 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:04.600237 (+ 2508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:04.602832 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:04.604161 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:04.604376 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:04.609802 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:04.609814 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:04.609838 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:04.609901 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:04.614457 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:04.615574 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:04.615585 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:04.615596 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:04.615812 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:04.616379 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:04.616384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:04.616386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:04.617172 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:04.617509 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:04.618041 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 09:07:04.618505 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11459,"thread_start_us":11916,"threads_started":1}
I20260806 09:07:04.622793 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:04.623363 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd351b4e2fa46339d8a0b55d6db324d; no voters:
I20260806 09:07:04.624105 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:07:04.624246 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:04.625650 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 LEADER]: Becoming Leader. State: Replica: 9bd351b4e2fa46339d8a0b55d6db324d, State: Running, Role: LEADER
I20260806 09:07:04.626358 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:04.626552 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:04.632784 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } } }
I20260806 09:07:04.633257 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:04.633832 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd351b4e2fa46339d8a0b55d6db324d. Latest consensus state: current_term: 1 leader_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } } }
I20260806 09:07:04.634150 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:04.640699 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:04.647921 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:04.649047 8639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:07:04.684327 8651 catalog_manager.cc:1347] Generated new cluster ID: cbcb5efcff464e5cb407fcba2a935a7c
I20260806 09:07:04.684489 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:04.725561 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:07:04.727892 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:04.739147 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Generated new TSK 0
I20260806 09:07:04.740512 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:07:05.673115 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:05 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007225, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:07:06.155328 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:06.155877 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:06.156104 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:06.168383 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:06.168654 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:06.168900 8684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:06.185601 8684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:06.187786 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:06.189638 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:06.208788 8694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:06.209846 8695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:06.216534 8697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:06.216553 8684 server_base.cc:1034] running on GCE node
I20260806 09:07:06.217484 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:06.218802 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:06.220150 8684 hybrid_clock.cc:648] HybridClock initialized: now 1786007226220071 us; error 83 us; skew 500 ppm
I20260806 09:07:06.220615 8684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:06 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007226, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:06.224493 8684 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:06.226567 8684 webserver.cc:460] Webserver started at http://127.27.187.129:43883/ using document root <none> and password file <none>
I20260806 09:07:06.227545 8684 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:06.227694 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:06.228060 8684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:07:06 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007226, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:06.265564 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "7f3d4f5720034844ae2bf402350c5193"
format_stamp: "Formatted at 2026-08-06 09:07:06 on dist-test-slave-8tdl"
server_key: "01e8bf26b9bb8d00f48f6cdcf5208b77"
server_key_iv: "d734317f9cda65273b4757be18d6e581"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:06.267305 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "7f3d4f5720034844ae2bf402350c5193"
format_stamp: "Formatted at 2026-08-06 09:07:06 on dist-test-slave-8tdl"
server_key: "01e8bf26b9bb8d00f48f6cdcf5208b77"
server_key_iv: "d734317f9cda65273b4757be18d6e581"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:06.278183 8684 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.007s
I20260806 09:07:06.311436 8704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:06.320571 8684 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.006s sys 0.009s
I20260806 09:07:06.320937 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "7f3d4f5720034844ae2bf402350c5193"
format_stamp: "Formatted at 2026-08-06 09:07:06 on dist-test-slave-8tdl"
server_key: "01e8bf26b9bb8d00f48f6cdcf5208b77"
server_key_iv: "d734317f9cda65273b4757be18d6e581"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:06.321363 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:06.369112 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:06.375245 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:06.375707 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:06.383297 8684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:06.392072 8684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:06.392288 8684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 09:07:06.392503 8684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:06.392632 8684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:07:06 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007225, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:06.634258 8684 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:35411
I20260806 09:07:06.634292 8816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:35411 every 8 connection(s)
I20260806 09:07:06.636158 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:06.638564 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 8684
I20260806 09:07:06.638824 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:07:06.639700 28398 external_mini_cluster.cc:1428] Setting key 2bc2950c9391a72adea546f6df0aa15d
Aug 06 09:07:06 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007226, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
I20260806 09:07:06.682955 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:06.640916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:58353 (local address 127.27.187.190:45283)
0806 09:07:06.657154 (+ 16238us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:06.657162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:06.663552 (+ 6390us) server_negotiation.cc:407] Connection header received
0806 09:07:06.663639 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:06.663644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:06.663748 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:06.663858 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:06.666293 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:06.667496 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:06.670041 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:06.670708 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:06.670878 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:06.675882 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:06.675890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:06.675897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:06.675951 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:06.679282 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:06.680208 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:06.680214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:06.680224 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:06.680382 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:06.680917 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:06.680924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:06.680927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:06.681231 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:06.681392 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:06.682270 (+ 878us) server_negotiation.cc:299] Negotiation successful
0806 09:07:06.682510 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16064,"thread_start_us":16255,"threads_started":1}
I20260806 09:07:06.684113 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:06.640036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.129:58353)
0806 09:07:06.659835 (+ 19799us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:06.659912 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:06.662417 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:06.664417 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:06.664429 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:06.665124 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:06.666005 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:06.666034 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:06.667773 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:06.667778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:06.669771 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:06.669799 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:06.671022 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:06.671033 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:06.671391 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:06.672353 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:06.672392 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:06.675509 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:06.679493 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:06.679505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:06.679527 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:06.680058 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:06.680522 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:06.680527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:06.680531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:06.680792 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:06.681398 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:06.681410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:06.681854 (+ 444us) client_negotiation.cc:769] Sending connection context
0806 09:07:06.682190 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 09:07:06.682900 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19426,"thread_start_us":19150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
I20260806 09:07:06.686126 8817 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:06.686486 8817 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:06.687507 8817 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:06.690397 8604 ts_manager.cc:194] Registered new tserver with Master: 7f3d4f5720034844ae2bf402350c5193 (127.27.187.129:35411)
I20260806 09:07:06.692975 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:58353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:07:06.727406 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:06 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007225, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
W20260806 09:07:07.265720 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:07.266252 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:07.266441 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:07.277748 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:07.278008 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:07.278252 8823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:07.293622 8823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:07.295835 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:07.297652 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:07.316020 8837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:07.317502 8838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:07.324239 8840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:07.324465 8823 server_base.cc:1034] running on GCE node
I20260806 09:07:07.325307 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:07.326653 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:07.328058 8823 hybrid_clock.cc:648] HybridClock initialized: now 1786007227327989 us; error 59 us; skew 500 ppm
I20260806 09:07:07.328480 8823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:07 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:07.332976 8823 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:07.335189 8823 webserver.cc:460] Webserver started at http://127.27.187.130:33975/ using document root <none> and password file <none>
I20260806 09:07:07.336220 8823 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:07.336371 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:07.336737 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:07:07 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:07.357807 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "9a709b79ccea4386b15c8f7235f6413a"
format_stamp: "Formatted at 2026-08-06 09:07:07 on dist-test-slave-8tdl"
server_key: "e6ae48c742506047ecebb79742c34fa5"
server_key_iv: "b11508fe06419769c66cd322d934c1bc"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:07.359079 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "9a709b79ccea4386b15c8f7235f6413a"
format_stamp: "Formatted at 2026-08-06 09:07:07 on dist-test-slave-8tdl"
server_key: "e6ae48c742506047ecebb79742c34fa5"
server_key_iv: "b11508fe06419769c66cd322d934c1bc"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:07.367138 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260806 09:07:07.387385 8847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:07.389540 8823 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:07:07.389773 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "9a709b79ccea4386b15c8f7235f6413a"
format_stamp: "Formatted at 2026-08-06 09:07:07 on dist-test-slave-8tdl"
server_key: "e6ae48c742506047ecebb79742c34fa5"
server_key_iv: "b11508fe06419769c66cd322d934c1bc"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:07.390080 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:07.438784 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:07.445101 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:07.445488 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:07.446856 8823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:07.448830 8823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:07.448935 8823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:07.449069 8823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:07.449136 8823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:07.492575 8823 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36529
I20260806 09:07:07.492628 8959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36529 every 8 connection(s)
I20260806 09:07:07.494354 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:07:07.500759 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 8823
I20260806 09:07:07.500952 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:07:07.501480 28398 external_mini_cluster.cc:1428] Setting key cc8462ed687a4a6dc6c19dbd68e9658f
Aug 06 09:07:07 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:07.542129 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:07.497648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:46867 (local address 127.27.187.190:45283)
0806 09:07:07.514167 (+ 16519us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:07.514174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:07.516415 (+ 2241us) server_negotiation.cc:407] Connection header received
0806 09:07:07.519339 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:07.519346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:07.519440 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:07.519522 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:07.522399 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:07.523653 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:07.526530 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:07.527228 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:07.527357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:07.532465 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:07.532473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:07.532479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:07.532531 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:07.536401 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:07.537558 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:07.537564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:07.537568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:07.537736 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:07.538373 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:07.538378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:07.538382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:07.538792 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:07.538929 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:07.541443 (+ 2514us) server_negotiation.cc:299] Negotiation successful
0806 09:07:07.541743 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16339,"thread_start_us":16530,"threads_started":1}
I20260806 09:07:07.543242 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:07.498347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.130:46867)
0806 09:07:07.516212 (+ 17865us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:07.516289 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:07.518850 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:07.520172 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:07.520186 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:07.520910 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:07.522075 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:07.522108 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:07.523939 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:07.523945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:07.526281 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:07.526310 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:07.527798 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:07.527808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:07.528162 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:07.529075 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:07.529119 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:07.532155 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:07.536582 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:07.536594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:07.536614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:07.537402 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:07.537890 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:07.537896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:07.537900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:07.538221 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:07.539029 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:07.539038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:07.539607 (+ 569us) client_negotiation.cc:769] Sending connection context
0806 09:07:07.539943 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 09:07:07.540536 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17469,"thread_start_us":29790,"threads_started":1}
I20260806 09:07:07.544966 8960 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:07.545389 8960 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:07.546321 8960 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:07.548365 8604 ts_manager.cc:194] Registered new tserver with Master: 9a709b79ccea4386b15c8f7235f6413a (127.27.187.130:36529)
I20260806 09:07:07.549680 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:46867
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:07:07.598841 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:07.696887 8817 heartbeater.cc:503] Master 127.27.187.190:45283 was elected leader, sending a full tablet report...
W20260806 09:07:08.036715 8966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:08.037106 8966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:08.037302 8966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:08.045535 8966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:08.045675 8966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:08.045849 8966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:08.056375 8966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:08.057930 8966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:08.059342 8966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:08.073901 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:08.073856 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:08.073930 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:08.075068 8966 server_base.cc:1034] running on GCE node
I20260806 09:07:08.075726 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:08.076797 8966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:08.078029 8966 hybrid_clock.cc:648] HybridClock initialized: now 1786007228077957 us; error 62 us; skew 500 ppm
I20260806 09:07:08.078444 8966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:08.082412 8966 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:08.084216 8966 webserver.cc:460] Webserver started at http://127.27.187.131:45571/ using document root <none> and password file <none>
I20260806 09:07:08.085073 8966 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:08.085192 8966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:08.085625 8966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:08.102370 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "10ad6820c4b044028bb584751a5af9ef"
format_stamp: "Formatted at 2026-08-06 09:07:08 on dist-test-slave-8tdl"
server_key: "8735207be8793a1b72209342102bfaf2"
server_key_iv: "fbf2aa2b1987e0e5d402a95fc57a0716"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:08.103386 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "10ad6820c4b044028bb584751a5af9ef"
format_stamp: "Formatted at 2026-08-06 09:07:08 on dist-test-slave-8tdl"
server_key: "8735207be8793a1b72209342102bfaf2"
server_key_iv: "fbf2aa2b1987e0e5d402a95fc57a0716"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:08.109475 8966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 09:07:08.128304 8981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:08.130571 8966 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:07:08.130849 8966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "10ad6820c4b044028bb584751a5af9ef"
format_stamp: "Formatted at 2026-08-06 09:07:08 on dist-test-slave-8tdl"
server_key: "8735207be8793a1b72209342102bfaf2"
server_key_iv: "fbf2aa2b1987e0e5d402a95fc57a0716"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:08.131086 8966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:08.144557 8966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:08.148960 8966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:08.149319 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:08.150386 8966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:08.152093 8966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:08.152173 8966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:08.152315 8966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:08.152366 8966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:08.195725 8966 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:35393
I20260806 09:07:08.195793 9094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:35393 every 8 connection(s)
I20260806 09:07:08.197278 8966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:08.200697 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 8966
I20260806 09:07:08.200945 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:07:08.201594 28398 external_mini_cluster.cc:1428] Setting key ad1f0a51c2531031580ab9683a01d0d8
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:08.235796 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:08.201645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:37871 (local address 127.27.187.190:45283)
0806 09:07:08.215277 (+ 13632us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:08.215284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:08.215701 (+ 417us) server_negotiation.cc:407] Connection header received
0806 09:07:08.218400 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:08.218407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:08.218506 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:08.218654 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:08.220665 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.221492 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.223440 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.223909 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.224056 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:08.228300 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:08.228307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:08.228311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:08.228388 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:08.232028 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.233088 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.233093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.233095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.233240 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.233745 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.233748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.233750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.234040 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:08.234162 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:08.235288 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0806 09:07:08.235518 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13475,"thread_start_us":13279,"threads_started":1}
I20260806 09:07:08.237177 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:08.202420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.131:37871)
0806 09:07:08.215489 (+ 13069us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:08.215541 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:08.217985 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:08.219053 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:08.219066 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:08.219762 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:08.220466 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:08.220488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.221674 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.221680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:08.223237 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:08.223254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.224161 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.224171 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:08.224475 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:08.225244 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:08.225342 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:08.227983 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:08.232230 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:08.232241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:08.232261 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:08.232950 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:08.233347 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:08.233353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:08.233357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:08.233619 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:08.234170 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:08.234178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:08.234669 (+ 491us) client_negotiation.cc:769] Sending connection context
0806 09:07:08.235040 (+ 371us) client_negotiation.cc:240] Negotiation successful
0806 09:07:08.235645 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12755,"thread_start_us":12387,"threads_started":1}
I20260806 09:07:08.238834 9095 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:08.239246 9095 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:08.240064 9095 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:08.241824 8604 ts_manager.cc:194] Registered new tserver with Master: 10ad6820c4b044028bb584751a5af9ef (127.27.187.131:35393)
I20260806 09:07:08.243094 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:37871
I20260806 09:07:08.245535 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:07:08.247615 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 8684
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:07:08.266827 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:08.248767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.27.187.129:35411)
0806 09:07:08.249499 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:08.249508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:08.249542 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:07:08.249612 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:08.249620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:08.249907 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:08.250088 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:08.251811 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.253009 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.254723 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.255421 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.255531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:08.260368 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:08.260385 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:08.260401 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:08.260442 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:08.263701 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.264531 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.264540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.264542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.264708 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.265136 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.265140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.265142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.265577 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:08.265847 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:08.266150 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 09:07:08.266468 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":378,"threads_started":1}
I20260806 09:07:08.268137 8812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:08.353989 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 8684
I20260806 09:07:08.373620 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 8823
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:07:08.392382 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:08.374854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.27.187.130:36529)
0806 09:07:08.375498 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:08.375507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:08.375537 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:07:08.375605 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:08.375612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:08.375844 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:08.375993 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:08.377363 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.378566 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.380305 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.380928 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.381042 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:08.386012 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:08.386026 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:08.386042 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:08.386091 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:08.389427 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.390216 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.390223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.390225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.390416 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.390885 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.390889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.390891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.391304 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:08.391552 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:08.391849 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 09:07:08.392079 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":328,"threads_started":1}
I20260806 09:07:08.393775 8955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:08.481935 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 8823
I20260806 09:07:08.501147 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 8966
Aug 06 09:07:08 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:07:08.520833 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:08.502370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.27.187.131:35393)
0806 09:07:08.503127 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:08.503136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:08.503244 (+ 108us) server_negotiation.cc:407] Connection header received
0806 09:07:08.503318 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:08.503325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:08.503556 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:08.503772 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:08.505257 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.506325 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.507840 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:08.508525 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:08.508678 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:08.513945 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:08.513961 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:08.513978 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:08.514053 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:08.517280 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.518212 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.518221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.518223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.518389 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:08.518877 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:08.518883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:08.518886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:08.519283 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:08.519719 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:08.520094 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 09:07:08.520440 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":432,"threads_started":1}
I20260806 09:07:08.522390 9090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:08.609755 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 8966
I20260806 09:07:08.629652 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 8572
I20260806 09:07:08.631201 8634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:08.740830 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 8572
I20260806 09:07:08.765195 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45283
--webserver_interface=127.27.187.190
--webserver_port=36787
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45283
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:09.134939 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:09.135277 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:09.135423 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:09.144520 9107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:09.144667 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:09.144752 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:09.144861 9107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:09.144944 9107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:09.155826 9107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45283
--ranger_config_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=36787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:09.157399 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:09.158964 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:09.171944 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:09.171875 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:09.171900 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:09.172822 9107 server_base.cc:1034] running on GCE node
I20260806 09:07:09.173709 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:09.175465 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:09.176743 9107 hybrid_clock.cc:648] HybridClock initialized: now 1786007229176676 us; error 61 us; skew 500 ppm
I20260806 09:07:09.177163 9107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:09 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007229, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:09.180979 9107 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:09.182765 9107 webserver.cc:460] Webserver started at http://127.27.187.190:36787/ using document root <none> and password file <none>
I20260806 09:07:09.183483 9107 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:09.183638 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:07:09 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007229, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:09.253355 9107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:07:09.256919 9123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:09.258615 9107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:07:09.258909 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "9bd351b4e2fa46339d8a0b55d6db324d"
format_stamp: "Formatted at 2026-08-06 09:07:04 on dist-test-slave-8tdl"
server_key: "37c99f43776fe12bb8d8d36a8c5e9599"
server_key_iv: "a9b87a2054e3a1d7bd42c4936f7969f8"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:09.259974 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:09.285079 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:09.290203 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:09.290580 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:09.312242 9107 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45283
I20260806 09:07:09.312305 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45283 every 8 connection(s)
I20260806 09:07:09.313968 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:09.319001 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9107
I20260806 09:07:09.320128 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:35411
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:45283
--webserver_port=43883
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:09.322458 9176 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:07:09.328128 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Bootstrap starting.
I20260806 09:07:09.349809 9176 log.cc:824] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:09.361094 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:07:09.361769 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Bootstrap complete.
I20260806 09:07:09.373132 9176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:09.374176 9176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd351b4e2fa46339d8a0b55d6db324d, State: Initialized, Role: FOLLOWER
I20260806 09:07:09.374944 9176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:09.375218 9176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:09.375360 9176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:09.375588 9176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:07:09.380182 9176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:09.380813 9176 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd351b4e2fa46339d8a0b55d6db324d; no voters:
I20260806 09:07:09.381552 9176 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:07:09.381718 9180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:07:09.383129 9180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [term 2 LEADER]: Becoming Leader. State: Replica: 9bd351b4e2fa46339d8a0b55d6db324d, State: Running, Role: LEADER
I20260806 09:07:09.383980 9180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } }
I20260806 09:07:09.384387 9176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:09.387008 9181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } } }
I20260806 09:07:09.387444 9181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:09.387151 9182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd351b4e2fa46339d8a0b55d6db324d. Latest consensus state: current_term: 2 leader_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd351b4e2fa46339d8a0b55d6db324d" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45283 } } }
I20260806 09:07:09.387624 9182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:09.396166 9185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:09.402601 9185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:09.404223 9185 catalog_manager.cc:1259] Loaded cluster ID: cbcb5efcff464e5cb407fcba2a935a7c
I20260806 09:07:09.404431 9185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:09.408813 9185 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:09.410616 9185 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Loaded TSK: 0
I20260806 09:07:09.412540 9176 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:07:09.412742 9185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:07:09.833521 9178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:09.834085 9178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:09.834329 9178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:09.847139 9178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:09.847385 9178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:09.847667 9178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:09.864862 9178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:35411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=43883
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:09.866899 9178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:09.868741 9178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:09.889575 9213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:09.892233 9214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:09.896328 9216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:09.898393 9178 server_base.cc:1034] running on GCE node
I20260806 09:07:09.899384 9178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:09.905443 9178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:09.907732 9178 hybrid_clock.cc:648] HybridClock initialized: now 1786007229907595 us; error 159 us; skew 500 ppm
I20260806 09:07:09.908167 9178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:09 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007229, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:09.916780 9178 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:09.920460 9178 webserver.cc:460] Webserver started at http://127.27.187.129:43883/ using document root <none> and password file <none>
I20260806 09:07:09.921489 9178 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:09.921689 9178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:07:09 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007229, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:09.971815 9178 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.006s sys 0.009s
I20260806 09:07:09.976207 9224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:09.977960 9178 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:07:09.978196 9178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "7f3d4f5720034844ae2bf402350c5193"
format_stamp: "Formatted at 2026-08-06 09:07:06 on dist-test-slave-8tdl"
server_key: "01e8bf26b9bb8d00f48f6cdcf5208b77"
server_key_iv: "d734317f9cda65273b4757be18d6e581"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:09.979466 9178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:10.028362 9178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:10.033152 9178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:10.033522 9178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:10.034713 9178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:10.036403 9178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:10.036520 9178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:10.036651 9178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:10.036751 9178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:10.110427 9178 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:35411
I20260806 09:07:10.110464 9338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:35411 every 8 connection(s)
I20260806 09:07:10.112396 9178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:10.118615 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9178
I20260806 09:07:10.119714 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:36529
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:45283
--webserver_port=33975
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:10 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007229, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:10.152590 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:10.116704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.129:38349)
0806 09:07:10.121811 (+ 5107us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:10.121886 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:10.124530 (+ 2644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:10.127537 (+ 3007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:10.127551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:10.128288 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:10.129178 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:10.129210 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.131904 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.131913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:10.134246 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:10.134270 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.135972 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.135982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:10.136324 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:10.137248 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:10.137292 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:10.140513 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:10.146044 (+ 5531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:10.146057 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:10.146077 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:10.146900 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:10.147541 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:10.147549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:10.147555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:10.147823 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:10.149178 (+ 1355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:10.149189 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:10.149722 (+ 533us) client_negotiation.cc:769] Sending connection context
0806 09:07:10.150100 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 09:07:10.150837 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4718,"thread_start_us":4464,"threads_started":1}
I20260806 09:07:10.156306 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:10.117644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:38349 (local address 127.27.187.190:45283)
0806 09:07:10.125521 (+ 7877us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:10.125544 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:10.125612 (+ 68us) server_negotiation.cc:407] Connection header received
0806 09:07:10.126109 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:10.126137 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:10.126815 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:10.127306 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:10.129500 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.131637 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.134467 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.135662 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.135895 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:10.140966 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:10.140982 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:10.141008 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:10.141078 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:10.145764 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:10.147110 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:10.147123 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:10.147135 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:10.147375 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:10.147984 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:10.147993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:10.147999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:10.148837 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:10.149069 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:10.153513 (+ 4444us) server_negotiation.cc:299] Negotiation successful
0806 09:07:10.154122 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7561,"thread_start_us":7345,"threads_started":1}
I20260806 09:07:10.161574 9339 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:10.161942 9339 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:10.162904 9339 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
W20260806 09:07:10.167203 9339 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:45283 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:07:10.167498 9339 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:10.167872 9339 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:10.169504 9339 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:10.169866 9339 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
W20260806 09:07:10.171324 9339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:07:10.552948 9343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:10.553319 9343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:10.553504 9343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:10.562563 9343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:10.562783 9343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:10.562999 9343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:10.577299 9343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:36529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=33975
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:10.579295 9343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:10.580819 9343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:10.602556 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:10.604508 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:10.605718 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:10.606635 9343 server_base.cc:1034] running on GCE node
I20260806 09:07:10.607652 9343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:10.608960 9343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:10.610248 9343 hybrid_clock.cc:648] HybridClock initialized: now 1786007230610141 us; error 104 us; skew 500 ppm
I20260806 09:07:10.610635 9343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:10 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:10.615468 9343 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:10.620682 9343 webserver.cc:460] Webserver started at http://127.27.187.130:33975/ using document root <none> and password file <none>
I20260806 09:07:10.621815 9343 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:10.622002 9343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:07:10 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:10.662492 9343 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.008s
I20260806 09:07:10.683499 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:10.686030 9343 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.007s sys 0.012s
I20260806 09:07:10.686360 9343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "9a709b79ccea4386b15c8f7235f6413a"
format_stamp: "Formatted at 2026-08-06 09:07:07 on dist-test-slave-8tdl"
server_key: "e6ae48c742506047ecebb79742c34fa5"
server_key_iv: "b11508fe06419769c66cd322d934c1bc"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:10.688165 9343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:10.715569 9343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:10.721916 9343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:10.722323 9343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:10.730062 9343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:10.738713 9343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:10.738952 9343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:07:10.739127 9343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:10.739243 9343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:10.817397 9343 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36529
I20260806 09:07:10.817960 9483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36529 every 8 connection(s)
I20260806 09:07:10.819597 9343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
Aug 06 09:07:10 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:10.825336 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9343
I20260806 09:07:10.826407 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.187.148:58257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:35393
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:45283
--webserver_port=45571
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:10 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:10.878634 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:10.822765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:54185 (local address 127.27.187.190:45283)
0806 09:07:10.841620 (+ 18855us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:10.841630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:10.841663 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:07:10.847796 (+ 6133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:10.847804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:10.847931 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:10.848042 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:10.852319 (+ 4277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.853574 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.864097 (+ 10523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.864877 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.865060 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:10.870105 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:10.870113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:10.870119 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:10.870165 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:10.873774 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:10.874899 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:10.874906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:10.874910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:10.875068 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:10.875603 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:10.875609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:10.875612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:10.875980 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:10.876112 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:10.877932 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0806 09:07:10.878212 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18672,"thread_start_us":27926,"threads_started":1}
I20260806 09:07:10.881335 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:10.823570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.130:54185)
0806 09:07:10.830853 (+ 7283us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:10.830937 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:10.843409 (+ 12472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:10.848845 (+ 5436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:10.848861 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:10.849608 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:10.850625 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:10.850660 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.853907 (+ 3247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.853913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:10.856229 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:10.856251 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:10.865540 (+ 9289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:10.865548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:10.865950 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:10.866865 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:10.866900 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:10.869781 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:10.873999 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:10.874009 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:10.874031 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:10.874735 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:10.875218 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:10.875223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:10.875227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:10.875474 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:10.876214 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:10.876224 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:10.876682 (+ 458us) client_negotiation.cc:769] Sending connection context
0806 09:07:10.879326 (+ 2644us) client_negotiation.cc:240] Negotiation successful
0806 09:07:10.880002 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":6979,"threads_started":1}
I20260806 09:07:10.883383 9484 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:10.883735 9484 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:10.884560 9484 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:10.887109 9141 ts_manager.cc:194] Registered new tserver with Master: 9a709b79ccea4386b15c8f7235f6413a (127.27.187.130:36529)
I20260806 09:07:10.889281 9141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:54185
I20260806 09:07:11.171479 9339 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:11.171993 9339 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:11.173419 9141 ts_manager.cc:194] Registered new tserver with Master: 7f3d4f5720034844ae2bf402350c5193 (127.27.187.129:35411)
I20260806 09:07:11.174861 9141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:38349
Aug 06 09:07:11 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
W20260806 09:07:11.354578 9488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:11.355158 9488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:11.355378 9488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:11.367449 9488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:11.367625 9488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:11.367892 9488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:11.381361 9488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.187.148:58257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:35393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=45571
--tserver_master_addrs=127.27.187.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:11.382993 9488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:11.384497 9488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:11.403643 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:11.403631 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:11.408051 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:11.408325 9488 server_base.cc:1034] running on GCE node
I20260806 09:07:11.409299 9488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:11.410661 9488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:11.411983 9488 hybrid_clock.cc:648] HybridClock initialized: now 1786007231411917 us; error 60 us; skew 500 ppm
I20260806 09:07:11.412407 9488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:11 dist-test-slave-8tdl krb5kdc[8123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007231, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:11.416777 9488 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:11.418955 9488 webserver.cc:460] Webserver started at http://127.27.187.131:45571/ using document root <none> and password file <none>
I20260806 09:07:11.419983 9488 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:11.420174 9488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:07:11 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007231, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:11.454854 9488 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.005s
I20260806 09:07:11.458441 9505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:11.460230 9488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:07:11.460438 9488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "10ad6820c4b044028bb584751a5af9ef"
format_stamp: "Formatted at 2026-08-06 09:07:08 on dist-test-slave-8tdl"
server_key: "8735207be8793a1b72209342102bfaf2"
server_key_iv: "fbf2aa2b1987e0e5d402a95fc57a0716"
server_key_version: "kuduclusterkey@0"
I20260806 09:07:11.461542 9488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:11.498639 9488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:11.503407 9488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:11.503719 9488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:11.504856 9488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:11.506453 9488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:11.506575 9488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:11.506732 9488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:11.506820 9488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:11.563928 9488 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:35393
I20260806 09:07:11.564118 9619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:35393 every 8 connection(s)
I20260806 09:07:11.566246 9488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:11.574718 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9488
Aug 06 09:07:11 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007231, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:11.617624 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.570429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:34381 (local address 127.27.187.190:45283)
0806 09:07:11.587274 (+ 16845us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.587285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.589587 (+ 2302us) server_negotiation.cc:407] Connection header received
0806 09:07:11.593992 (+ 4405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.593998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.594115 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.594197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:11.596980 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.598406 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.600925 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.601583 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.601685 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.609367 (+ 7682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:11.609374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:11.609379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:11.609435 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:11.613124 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.614207 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.614213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.614216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.614368 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.614959 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.614964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.614967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.615329 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:11.615464 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.616624 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.617222 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":343,"server-negotiator.queue_time_us":16582,"thread_start_us":17474,"threads_started":1}
I20260806 09:07:11.618798 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.575921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.27.187.190:45283)
0806 09:07:11.588224 (+ 12303us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.588231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.588257 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:07:11.588335 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.588341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.588456 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.588617 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:11.592578 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.593708 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.596355 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.597570 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.597710 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.601813 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:11.601821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:11.601825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:11.601880 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:11.606960 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.607994 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.608000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.608004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.608487 (+ 483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.609751 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.609757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.609760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.610598 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:11.610997 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.615681 (+ 4684us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.618201 (+ 2520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12149,"thread_start_us":10951,"threads_started":1}
I20260806 09:07:11.618958 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.569851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45283 (local address 127.27.187.131:34381)
0806 09:07:11.589394 (+ 19543us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.589464 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.592180 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.595063 (+ 2883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.595079 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.595744 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:11.596696 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.596727 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.598646 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.598652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.600727 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.600744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.602070 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.602076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.602384 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.604211 (+ 1827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:11.604252 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:11.608939 (+ 4687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:11.613341 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:11.613352 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:11.613373 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:11.614041 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:11.614533 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:11.614537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:11.614540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:11.614830 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:11.615565 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:11.615573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:11.616081 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.616783 (+ 702us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.617483 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19201,"thread_start_us":18993,"threads_started":1}
I20260806 09:07:11.620903 9620 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45283
I20260806 09:07:11.621287 9620 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:11.622318 9620 heartbeater.cc:511] Master 127.27.187.190:45283 requested a full tablet report, sending...
I20260806 09:07:11.624770 9140 ts_manager.cc:194] Registered new tserver with Master: 10ad6820c4b044028bb584751a5af9ef (127.27.187.131:35393)
I20260806 09:07:11.626154 9140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:34381
I20260806 09:07:11.638358 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:07:11 dist-test-slave-8tdl krb5kdc[8123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for HTTP/127.27.187.148@KRBTEST.COM
I20260806 09:07:11.673512 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.654299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.27.187.190:45283)
0806 09:07:11.654619 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.654626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.655650 (+ 1024us) server_negotiation.cc:407] Connection header received
0806 09:07:11.655970 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.655977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.656077 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.656210 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:11.658291 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.659530 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.661800 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.662375 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.662467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.666543 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:11.666551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:11.666557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:11.666620 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:11.669878 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.670700 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.670708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.670714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.670899 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:11.671485 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:11.671490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:11.671493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:11.671907 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:11.672173 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.672918 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.673196 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260806 09:07:11.680258 9140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:07:11.691560 9140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:07:11.784873 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.738408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.27.187.130:36529)
0806 09:07:11.773284 (+ 34876us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.773299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.774260 (+ 961us) server_negotiation.cc:407] Connection header received
0806 09:07:11.774508 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.774518 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.774888 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.775144 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:11.776393 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.777921 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.780927 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.782511 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.782712 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.782935 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.784366 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.784559 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34596,"thread_start_us":34893,"threads_started":1}
I20260806 09:07:11.785995 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.736417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36529 (local address 127.0.0.1:52512)
0806 09:07:11.774090 (+ 37673us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.774117 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.774312 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.775525 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.775533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.775568 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:11.776126 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.776146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.778102 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.778107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.780703 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.780731 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.783197 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.783204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.783437 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.783505 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.783691 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.783928 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37529,"thread_start_us":24633,"threads_started":1}
I20260806 09:07:11.793351 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.734145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43074)
0806 09:07:11.760531 (+ 26386us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.760593 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.760816 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.783103 (+ 22287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.783114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.783170 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:11.783639 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.783654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.785695 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.785700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.787870 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.787886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.792013 (+ 4127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.792023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.792272 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.792313 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.792395 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.792552 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26044,"thread_start_us":39629,"threads_started":1}
I20260806 09:07:11.795019 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.762072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:35393 (local address 127.0.0.1:38810)
0806 09:07:11.783969 (+ 21897us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.783989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.784129 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.787166 (+ 3037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.787171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.787200 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:11.787592 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.787604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.789756 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.789762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.791926 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.791944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.794177 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.794184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.794362 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.794396 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.794482 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.794616 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21768,"thread_start_us":27529,"threads_started":1}
I20260806 09:07:11.795125 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.735001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.27.187.129:35411)
0806 09:07:11.777003 (+ 42002us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.777013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.777044 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:07:11.782360 (+ 5316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.782371 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.782793 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.783052 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:11.783913 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.785523 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.788116 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.789649 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.794156 (+ 4507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.794373 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.794453 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.794769 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41795,"thread_start_us":48254,"threads_started":1}
I20260806 09:07:11.795460 9419 tablet_service.cc:1467] Processing CreateTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 (DEFAULT_TABLE table=test-table [id=622461ddb75e459b981ae2b6e416757c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:11.797268 9419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d995d70ea184cfa8b2f509cc9677d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:11.799757 9274 tablet_service.cc:1467] Processing CreateTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 (DEFAULT_TABLE table=test-table [id=622461ddb75e459b981ae2b6e416757c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:11.801908 9274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d995d70ea184cfa8b2f509cc9677d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:11.796561 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.767856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.27.187.131:35393)
0806 09:07:11.783741 (+ 15885us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.783752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.784133 (+ 381us) server_negotiation.cc:407] Connection header received
0806 09:07:11.786639 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.786646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.786947 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.787179 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:11.787846 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.789573 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.792181 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.793759 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.795730 (+ 1971us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.795936 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.796008 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.796192 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15659,"thread_start_us":15888,"threads_started":1}
I20260806 09:07:11.823127 9554 tablet_service.cc:1467] Processing CreateTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 (DEFAULT_TABLE table=test-table [id=622461ddb75e459b981ae2b6e416757c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:11.827854 9554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d995d70ea184cfa8b2f509cc9677d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:11.840561 9642 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: Bootstrap starting.
I20260806 09:07:11.843739 9642 tablet_bootstrap.cc:654] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:11.845192 9642 log.cc:824] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:11.868340 9642 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: No bootstrap required, opened a new log
I20260806 09:07:11.869189 9642 ts_tablet_manager.cc:1397] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.007s
I20260806 09:07:11.876614 9645 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Bootstrap starting.
I20260806 09:07:11.876614 9644 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Bootstrap starting.
I20260806 09:07:11.881044 9645 tablet_bootstrap.cc:654] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:11.882634 9645 log.cc:824] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:11.884153 9644 tablet_bootstrap.cc:654] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:11.892845 9645 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: No bootstrap required, opened a new log
I20260806 09:07:11.893286 9645 ts_tablet_manager.cc:1397] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20260806 09:07:11.894737 9484 heartbeater.cc:503] Master 127.27.187.190:45283 was elected leader, sending a full tablet report...
I20260806 09:07:11.897711 9644 log.cc:824] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:11.902776 9642 raft_consensus.cc:348] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.905181 9642 raft_consensus.cc:374] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:11.905402 9642 raft_consensus.cc:729] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3d4f5720034844ae2bf402350c5193, State: Initialized, Role: FOLLOWER
I20260806 09:07:11.905275 9645 raft_consensus.cc:348] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.906204 9645 raft_consensus.cc:374] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:11.906349 9645 raft_consensus.cc:729] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ad6820c4b044028bb584751a5af9ef, State: Initialized, Role: FOLLOWER
I20260806 09:07:11.906563 9642 consensus_queue.cc:260] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.907112 9645 consensus_queue.cc:260] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.912999 9645 ts_tablet_manager.cc:1428] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Time spent starting tablet: real 0.020s user 0.011s sys 0.005s
I20260806 09:07:11.913596 9620 heartbeater.cc:503] Master 127.27.187.190:45283 was elected leader, sending a full tablet report...
I20260806 09:07:11.913836 9644 tablet_bootstrap.cc:492] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: No bootstrap required, opened a new log
I20260806 09:07:11.914165 9644 ts_tablet_manager.cc:1397] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.003s
I20260806 09:07:11.914327 9339 heartbeater.cc:503] Master 127.27.187.190:45283 was elected leader, sending a full tablet report...
I20260806 09:07:11.926903 9644 raft_consensus.cc:348] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.927573 9644 raft_consensus.cc:374] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:11.927717 9644 raft_consensus.cc:729] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a709b79ccea4386b15c8f7235f6413a, State: Initialized, Role: FOLLOWER
I20260806 09:07:11.928287 9649 raft_consensus.cc:482] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:07:11.928612 9649 raft_consensus.cc:504] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.928735 9644 consensus_queue.cc:260] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.929350 9642 ts_tablet_manager.cc:1428] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: Time spent starting tablet: real 0.060s user 0.016s sys 0.004s
I20260806 09:07:11.931116 9649 leader_election.cc:288] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f3d4f5720034844ae2bf402350c5193 (127.27.187.129:35411), 9a709b79ccea4386b15c8f7235f6413a (127.27.187.130:36529)
I20260806 09:07:11.940647 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.931492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.27.187.131:49525)
0806 09:07:11.932249 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.932269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.932390 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.933439 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.933445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.933479 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:11.933726 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.933740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.935642 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.935649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.937909 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.937931 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.939803 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.939811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.940035 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.940073 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.940182 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.940336 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627}
I20260806 09:07:11.941479 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.932537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49525 (local address 127.27.187.129:35411)
0806 09:07:11.932768 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.932774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.932796 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:07:11.932873 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.932879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.932961 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.933062 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:11.934069 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.935236 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.938172 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.939400 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.939589 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.939688 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.941050 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.941228 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:07:11.943018 9294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d995d70ea184cfa8b2f509cc9677d41" candidate_uuid: "10ad6820c4b044028bb584751a5af9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f3d4f5720034844ae2bf402350c5193" is_pre_election: true
I20260806 09:07:11.943660 9294 raft_consensus.cc:2454] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ad6820c4b044028bb584751a5af9ef in term 0.
I20260806 09:07:11.944684 9507 leader_election.cc:302] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ad6820c4b044028bb584751a5af9ef, 7f3d4f5720034844ae2bf402350c5193; no voters:
I20260806 09:07:11.945266 9649 raft_consensus.cc:2781] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:07:11.945417 9649 raft_consensus.cc:482] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:07:11.945510 9649 raft_consensus.cc:3037] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:11.948975 9649 raft_consensus.cc:504] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.949741 9649 leader_election.cc:288] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [CANDIDATE]: Term 1 election: Requested vote from peers 7f3d4f5720034844ae2bf402350c5193 (127.27.187.129:35411), 9a709b79ccea4386b15c8f7235f6413a (127.27.187.130:36529)
I20260806 09:07:11.950739 9294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d995d70ea184cfa8b2f509cc9677d41" candidate_uuid: "10ad6820c4b044028bb584751a5af9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f3d4f5720034844ae2bf402350c5193"
I20260806 09:07:11.950980 9294 raft_consensus.cc:3037] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:11.951035 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.932819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:57307 (local address 127.27.187.130:36529)
0806 09:07:11.933002 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:11.933008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:11.942252 (+ 9244us) server_negotiation.cc:407] Connection header received
0806 09:07:11.942571 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:11.942578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:11.942696 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:11.942765 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:11.943402 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.944483 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.946986 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.948408 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.949728 (+ 1320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.949846 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:11.950563 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 09:07:11.950754 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:07:11.951846 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:11.931919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36529 (local address 127.27.187.131:57307)
0806 09:07:11.942139 (+ 10220us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:11.942160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:11.942378 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:11.942986 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:11.942991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:11.943019 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:11.943221 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.943233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.944711 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.944716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.946743 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:11.946761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:11.950039 (+ 3278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:11.950049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:11.950329 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:11.950382 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:07:11.951375 (+ 993us) client_negotiation.cc:240] Negotiation successful
0806 09:07:11.951538 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10106,"threads_started":1}
I20260806 09:07:11.954372 9294 raft_consensus.cc:2454] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ad6820c4b044028bb584751a5af9ef in term 1.
I20260806 09:07:11.955260 9507 leader_election.cc:302] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ad6820c4b044028bb584751a5af9ef, 7f3d4f5720034844ae2bf402350c5193; no voters:
I20260806 09:07:11.955699 9649 raft_consensus.cc:2781] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:11.957453 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d995d70ea184cfa8b2f509cc9677d41" candidate_uuid: "10ad6820c4b044028bb584751a5af9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a709b79ccea4386b15c8f7235f6413a" is_pre_election: true
I20260806 09:07:11.958055 9439 raft_consensus.cc:2454] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ad6820c4b044028bb584751a5af9ef in term 0.
I20260806 09:07:11.959195 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d995d70ea184cfa8b2f509cc9677d41" candidate_uuid: "10ad6820c4b044028bb584751a5af9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a709b79ccea4386b15c8f7235f6413a"
I20260806 09:07:11.959407 9439 raft_consensus.cc:3037] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:11.962638 9439 raft_consensus.cc:2454] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ad6820c4b044028bb584751a5af9ef in term 1.
I20260806 09:07:11.969784 9649 raft_consensus.cc:686] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 1 LEADER]: Becoming Leader. State: Replica: 10ad6820c4b044028bb584751a5af9ef, State: Running, Role: LEADER
I20260806 09:07:11.970557 9649 consensus_queue.cc:237] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } }
I20260806 09:07:11.975159 9644 ts_tablet_manager.cc:1428] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Time spent starting tablet: real 0.061s user 0.022s sys 0.001s
I20260806 09:07:11.985236 9140 catalog_manager.cc:5495] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ad6820c4b044028bb584751a5af9ef (127.27.187.131). New cstate: current_term: 1 leader_uuid: "10ad6820c4b044028bb584751a5af9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10ad6820c4b044028bb584751a5af9ef" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 35393 } health_report { overall_health: HEALTHY } } }
I20260806 09:07:12.015808 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.006837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.27.187.131:35393)
0806 09:07:12.007161 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:12.007171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:12.007193 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:07:12.007317 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:12.007322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:12.007418 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:12.007589 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:07:12.008060 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.009479 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.011924 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.012643 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.012784 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:12.013557 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:07:12.014376 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:07:12.015298 (+ 922us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:12.015358 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 09:07:12.015461 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:07:12.027936 9294 raft_consensus.cc:1261] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 1 FOLLOWER]: Refusing update from remote peer 10ad6820c4b044028bb584751a5af9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:12.028015 9439 raft_consensus.cc:1261] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 1 FOLLOWER]: Refusing update from remote peer 10ad6820c4b044028bb584751a5af9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:12.028992 9649 consensus_queue.cc:1035] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3d4f5720034844ae2bf402350c5193" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 35411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:12.029811 9652 consensus_queue.cc:1035] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a709b79ccea4386b15c8f7235f6413a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:12.061344 9140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:12.061681 9140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:12.066378 9140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bd351b4e2fa46339d8a0b55d6db324d: Sending DeleteTablet for 3 replicas of tablet 8d995d70ea184cfa8b2f509cc9677d41
I20260806 09:07:12.067936 9274 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:12 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:43074
I20260806 09:07:12.068012 9419 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:12 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:52512
I20260806 09:07:12.068768 9554 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d995d70ea184cfa8b2f509cc9677d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:12 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:38810
I20260806 09:07:12.075030 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9178
I20260806 09:07:12.082000 9673 tablet_replica.cc:330] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: stopping tablet replica
I20260806 09:07:12.082772 9673 raft_consensus.cc:2229] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:12.083534 9673 raft_consensus.cc:2258] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:12.087033 9673 ts_tablet_manager.cc:1905] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:12.096026 9673 ts_tablet_manager.cc:1918] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:12.096379 9673 log.cc:1196] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/8d995d70ea184cfa8b2f509cc9677d41
I20260806 09:07:12.097136 9673 ts_tablet_manager.cc:1939] T 8d995d70ea184cfa8b2f509cc9677d41 P 9a709b79ccea4386b15c8f7235f6413a: Deleting consensus metadata
I20260806 09:07:12.099843 9124 catalog_manager.cc:4841] TS 9a709b79ccea4386b15c8f7235f6413a (127.27.187.130:36529): tablet 8d995d70ea184cfa8b2f509cc9677d41 (table test-table [id=622461ddb75e459b981ae2b6e416757c]) successfully deleted
I20260806 09:07:12.102303 9674 tablet_replica.cc:330] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: stopping tablet replica
I20260806 09:07:12.102895 9674 raft_consensus.cc:2229] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:07:12.103832 9674 raft_consensus.cc:2258] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:12.105960 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.078651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.27.187.129:35411)
0806 09:07:12.078908 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:12.078915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:12.080823 (+ 1908us) server_negotiation.cc:407] Connection header received
0806 09:07:12.080924 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:12.080932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:12.081049 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:12.081183 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:12.083440 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.084763 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.086863 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.087529 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.087682 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:12.095803 (+ 8121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:12.095829 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:12.095851 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:12.095913 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:12.100783 (+ 4870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.101814 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.101833 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.101838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.102056 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.102772 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.102778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.102781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.103448 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:12.105054 (+ 1606us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:12.105238 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 09:07:12.105516 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 09:07:12.106724 9674 ts_tablet_manager.cc:1905] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:12.107126 9334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:12.121717 9674 ts_tablet_manager.cc:1918] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:12.122009 9674 log.cc:1196] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/8d995d70ea184cfa8b2f509cc9677d41
I20260806 09:07:12.123027 9674 ts_tablet_manager.cc:1939] T 8d995d70ea184cfa8b2f509cc9677d41 P 10ad6820c4b044028bb584751a5af9ef: Deleting consensus metadata
I20260806 09:07:12.125571 9124 catalog_manager.cc:4841] TS 10ad6820c4b044028bb584751a5af9ef (127.27.187.131:35393): tablet 8d995d70ea184cfa8b2f509cc9677d41 (table test-table [id=622461ddb75e459b981ae2b6e416757c]) successfully deleted
I20260806 09:07:12.284719 9672 tablet_replica.cc:330] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193: stopping tablet replica
I20260806 09:07:12.285319 9672 raft_consensus.cc:2229] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:12.286101 9672 raft_consensus.cc:2258] T 8d995d70ea184cfa8b2f509cc9677d41 P 7f3d4f5720034844ae2bf402350c5193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:12.287314 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9178
W20260806 09:07:12.312597 9125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:35411 (error 108)
I20260806 09:07:12.313423 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9343
I20260806 09:07:12.329210 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.315843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43080)
0806 09:07:12.328814 (+ 12971us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.328998 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12796,"threads_started":1}
W20260806 09:07:12.329720 9125 catalog_manager.cc:4584] TS 7f3d4f5720034844ae2bf402350c5193 (127.27.187.129:35411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d995d70ea184cfa8b2f509cc9677d41: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
I20260806 09:07:12.333869 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.315248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.27.187.130:36529)
0806 09:07:12.315458 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:12.315464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:12.315487 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:07:12.315553 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:12.315558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:12.315656 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:12.315773 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:12.317649 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.318898 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.320869 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.321490 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.321608 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:12.325352 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:12.325373 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:12.325404 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:12.325462 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:12.330053 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.331132 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.331143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.331147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.331332 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.331909 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.331915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.331918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.332451 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:12.332780 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:12.333194 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 09:07:12.333458 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:07:12.334697 9479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:12.383698 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.383182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43088)
0806 09:07:12.383406 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.383541 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 09:07:12.384934 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.384498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43096)
0806 09:07:12.384672 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.384755 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 09:07:12.445608 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.445070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43112)
0806 09:07:12.445328 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.445434 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 09:07:12.447048 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.446593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43122)
0806 09:07:12.446802 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.446891 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:07:12.493649 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9343
I20260806 09:07:12.515014 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9488
I20260806 09:07:12.528034 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.527425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43128)
0806 09:07:12.527699 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.527816 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 09:07:12.529376 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.528796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43132)
0806 09:07:12.529069 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.529184 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 09:07:12.533324 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.515878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.27.187.131:35393)
0806 09:07:12.516117 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:12.516124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:12.516225 (+ 101us) server_negotiation.cc:407] Connection header received
0806 09:07:12.516454 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:12.516460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:12.516564 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:12.516706 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:12.518305 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.519369 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.520866 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:12.521386 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:12.521575 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:12.524802 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:12.524820 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:12.524838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:12.524895 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:12.529755 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.530645 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.530669 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.530721 (+ 52us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.530938 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:12.531393 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:12.531400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:12.531404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:12.531874 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:12.532260 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:12.532637 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 09:07:12.532905 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:07:12.534261 9615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:12.662094 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9488
I20260806 09:07:12.683410 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9107
I20260806 09:07:12.684844 9171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:12.804277 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:12.803690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:35411 (local address 127.0.0.1:43142)
0806 09:07:12.803928 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:12.804046 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:07:12.804661 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9107
I20260806 09:07:12.829433 28398 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:07:13.176489 28398 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:07:13.176704 28398 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:07:13.523315 28398 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:07:13.523 UTC [8153] LOG: received smart shutdown request
2026-08-06 09:07:13.527 UTC [8153] LOG: background worker "logical replication launcher" (PID 8162) exited with exit code 1
2026-08-06 09:07:13.527 UTC [8157] LOG: shutting down
2026-08-06 09:07:13.755 UTC [8153] LOG: database system is shut down
2026-08-06T09:07:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58102 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:07:14 dist-test-slave-8tdl krb5kdc[9689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:07:14 dist-test-slave-8tdl krb5kdc[9689](info): set up 2 sockets
Aug 06 09:07:14 dist-test-slave-8tdl krb5kdc[9689](info): commencing operation
krb5kdc: starting...
W20260806 09:07:15.079962 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:07:15 dist-test-slave-8tdl krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:07:15.411962 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:35519
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:15.785051 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:15.785497 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:15.785670 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:15.793687 9703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:15.793870 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:15.793985 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:15.794082 9703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:15.794165 9703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:15.804898 9703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:15.806643 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:15.808175 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:15.822435 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:15.822471 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:15.822500 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:15.823571 9703 server_base.cc:1034] running on GCE node
I20260806 09:07:15.824580 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:15.826314 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:15.827593 9703 hybrid_clock.cc:648] HybridClock initialized: now 1786007235827540 us; error 44 us; skew 500 ppm
I20260806 09:07:15.828105 9703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:15 dist-test-slave-8tdl krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:15.832188 9703 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:15.834070 9703 webserver.cc:460] Webserver started at http://127.27.187.190:42703/ using document root <none> and password file <none>
I20260806 09:07:15.834982 9703 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:15.835134 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:15.835469 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:15.838838 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "036438acae414985959027521ab3ab40"
format_stamp: "Formatted at 2026-08-06 09:07:15 on dist-test-slave-8tdl"
server_key: "0ce0fbe09e79fd7f9e083a1297fb6a98"
server_key_iv: "ea6f53d8e630aed1fcf9d3e8054956fa"
server_key_version: "encryptionkey@0"
I20260806 09:07:15.839986 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "036438acae414985959027521ab3ab40"
format_stamp: "Formatted at 2026-08-06 09:07:15 on dist-test-slave-8tdl"
server_key: "0ce0fbe09e79fd7f9e083a1297fb6a98"
server_key_iv: "ea6f53d8e630aed1fcf9d3e8054956fa"
server_key_version: "encryptionkey@0"
I20260806 09:07:15.846933 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:07:15.851164 9718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:15.852890 9703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:07:15.853134 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "036438acae414985959027521ab3ab40"
format_stamp: "Formatted at 2026-08-06 09:07:15 on dist-test-slave-8tdl"
server_key: "0ce0fbe09e79fd7f9e083a1297fb6a98"
server_key_iv: "ea6f53d8e630aed1fcf9d3e8054956fa"
server_key_version: "encryptionkey@0"
I20260806 09:07:15.853379 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:15.878450 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:15.883508 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:15.883841 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:15.904495 9703 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:35519
I20260806 09:07:15.904536 9770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:35519 every 8 connection(s)
I20260806 09:07:15.905972 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:15.910424 9771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:15.913537 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9703
I20260806 09:07:15.913820 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:07:15.915205 28398 external_mini_cluster.cc:1428] Setting key 26cad1cab453d755b4221038bdd140b2
I20260806 09:07:15.919100 9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:15 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.189@KRBTEST.COM" created.
I20260806 09:07:15.949159 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:15.922830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.27.187.190:35519)
0806 09:07:15.925448 (+ 2618us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:15.925455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:15.925503 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:07:15.927599 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:15.927611 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:15.928034 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:15.928306 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:15.929776 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:15.931920 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:15.933562 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:15.934716 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:15.934976 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:15.939186 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:15.939200 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:15.939235 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:15.939287 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:15.943889 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:15.944880 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:15.944891 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:15.944895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:15.945143 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:15.945658 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:15.945665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:15.945671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:15.946421 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:15.946590 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:15.947729 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0806 09:07:15.948005 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2389,"spinlock_wait_cycles":64896,"thread_start_us":2106,"threads_started":1}
I20260806 09:07:15.949172 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:15.922830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35519 (local address 127.0.0.1:59416)
0806 09:07:15.923954 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:15.924029 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:15.927058 (+ 3029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:15.928383 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:15.928388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:15.929036 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:15.929458 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:15.929478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:15.932199 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:15.932203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:15.933386 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:15.933405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:15.935028 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:15.935034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:15.935259 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:15.936041 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:15.936073 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:15.938889 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:15.944112 (+ 5223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:15.944125 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:15.944137 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:15.944674 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:15.945334 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:15.945338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:15.945340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:15.945524 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:15.946603 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:15.946611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:15.947046 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 09:07:15.947324 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 09:07:15.947797 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":500,"threads_started":1}
I20260806 09:07:15.952781 9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:07:15.954305 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:15.953690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48412)
0806 09:07:15.953938 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:15.954114 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 09:07:15.954792 9719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
I20260806 09:07:15.956947 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:15.956383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48428)
0806 09:07:15.956619 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:15.956739 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:07:15.957254 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:44173: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
I20260806 09:07:15.988202 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:44173
--webserver_interface=127.27.187.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:16.006889 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } attempt: 1
I20260806 09:07:16.008297 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.007770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48442)
0806 09:07:16.008003 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.008136 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 09:07:16.009836 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.009350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48458)
0806 09:07:16.009534 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.009627 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 09:07:16.010049 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:44173: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
I20260806 09:07:16.078462 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } attempt: 2
I20260806 09:07:16.079823 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.079288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48466)
0806 09:07:16.079566 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.079664 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 09:07:16.081043 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.080614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48482)
0806 09:07:16.080800 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.080893 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 09:07:16.081436 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:44173: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
I20260806 09:07:16.172760 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } attempt: 3
I20260806 09:07:16.174106 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.173629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48490)
0806 09:07:16.173850 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.173957 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:07:16.175400 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.174965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48494)
0806 09:07:16.175140 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.175243 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:07:16.175611 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:44173: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
I20260806 09:07:16.318930 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } attempt: 4
I20260806 09:07:16.320209 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.319634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48506)
0806 09:07:16.319975 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.320067 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260806 09:07:16.321449 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.320997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48520)
0806 09:07:16.321166 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.321315 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:07:16.321647 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:44173: Network error: Client connection negotiation failed: client connection to 127.27.187.189:44173: connect: Connection refused (error 111)
W20260806 09:07:16.358541 9778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:16.359016 9778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:16.359202 9778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:16.367288 9778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:16.367421 9778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:16.367519 9778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:16.367609 9778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:16.367692 9778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:16.378082 9778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.187.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:16.379601 9778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:16.381011 9778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:16.393893 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:16.393959 9786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:16.393925 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:16.394986 9778 server_base.cc:1034] running on GCE node
I20260806 09:07:16.395908 9778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:16.397482 9778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:16.398736 9778 hybrid_clock.cc:648] HybridClock initialized: now 1786007236398627 us; error 64 us; skew 500 ppm
I20260806 09:07:16.399147 9778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:16 dist-test-slave-8tdl krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:16.402724 9778 init.cc:377] Logged in from keytab as kudu/127.27.187.189@KRBTEST.COM (short username kudu)
I20260806 09:07:16.404552 9778 webserver.cc:460] Webserver started at http://127.27.187.189:38093/ using document root <none> and password file <none>
I20260806 09:07:16.405318 9778 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:16.405468 9778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:16.405817 9778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:16.409199 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data/instance:
uuid: "66d70802ec5e4865bea7969f3620479b"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "115c85932f13fc51205371ef246941b4"
server_key_iv: "257664e71d1807d3ee77ab37caff2fda"
server_key_version: "encryptionkey@0"
I20260806 09:07:16.410113 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal/instance:
uuid: "66d70802ec5e4865bea7969f3620479b"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "115c85932f13fc51205371ef246941b4"
server_key_iv: "257664e71d1807d3ee77ab37caff2fda"
server_key_version: "encryptionkey@0"
I20260806 09:07:16.416244 9778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:07:16.420421 9793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:16.422026 9778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:07:16.422256 9778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "66d70802ec5e4865bea7969f3620479b"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "115c85932f13fc51205371ef246941b4"
server_key_iv: "257664e71d1807d3ee77ab37caff2fda"
server_key_version: "encryptionkey@0"
I20260806 09:07:16.422541 9778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:16.464536 9778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:16.469668 9778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:16.470003 9778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:16.491385 9778 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.189:44173
I20260806 09:07:16.491426 9845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.189:44173 every 8 connection(s)
I20260806 09:07:16.492919 9778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
I20260806 09:07:16.497215 9846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:16.501791 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9778
I20260806 09:07:16.502012 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-1/wal/instance
I20260806 09:07:16.502602 28398 external_mini_cluster.cc:1428] Setting key 3b76afb90539d67b0a795bc50e436b9e
I20260806 09:07:16.506992 9846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:16 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.188@KRBTEST.COM; defaulting to no policy
I20260806 09:07:16.536332 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.511239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.27.187.190:35519)
0806 09:07:16.511908 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:16.511916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:16.513239 (+ 1323us) server_negotiation.cc:407] Connection header received
0806 09:07:16.515991 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:16.515999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:16.516132 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:16.516288 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:16.518075 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.519597 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.521584 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.522494 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.522727 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.527166 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:16.527178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:16.527186 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:16.527258 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:16.531991 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.533068 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.533077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.533081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.533291 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.533797 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.533803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.533806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.534314 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:16.534503 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:16.535598 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0806 09:07:16.535877 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.188@KRBTEST.COM" created.
I20260806 09:07:16.537760 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.512012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35519 (local address 127.0.0.1:59424)
0806 09:07:16.513020 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.513083 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:16.515621 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:16.516545 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:16.516598 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:16.517216 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:16.517858 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.517877 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.519836 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.519841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.521387 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.521404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.522865 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.522873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.523125 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.524039 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:16.524075 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:16.526849 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:16.532311 (+ 5462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.532325 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.532341 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.532901 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.533453 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.533457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.533460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.533666 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.534518 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:16.534527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:16.535068 (+ 541us) client_negotiation.cc:769] Sending connection context
0806 09:07:16.535474 (+ 406us) client_negotiation.cc:240] Negotiation successful
0806 09:07:16.536187 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":348,"threads_started":1}
I20260806 09:07:16.539919 9846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:16 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:07:16.558699 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.540884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48532)
0806 09:07:16.541204 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.541226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:16.541358 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:16.542632 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:16.542636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:16.543237 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:16.543397 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.543406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.545749 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.545754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.546810 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.546823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.548447 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.548453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.548606 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.549361 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:16.549380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:16.551930 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:16.555855 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.555861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.555865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.556293 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.556702 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.556707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.556710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.556869 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.557635 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:16.557641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:16.557811 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 09:07:16.557951 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 09:07:16.558262 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:07:16.558800 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.541084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.27.187.189:44173)
0806 09:07:16.541968 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:16.541980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:16.542019 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:07:16.542102 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:16.542114 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:16.542411 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:16.542606 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:16.543573 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.545566 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.546970 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.548083 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.548252 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.552197 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:16.552212 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:16.552229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:16.552268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:16.555679 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.556447 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.556455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.556457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.556581 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.556980 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.556989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.556991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.557386 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:16.557740 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:16.558128 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 09:07:16.558374 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":515,"threads_started":1}
I20260806 09:07:16.561693 9846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:07:16.563052 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.562470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54526)
0806 09:07:16.562727 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.562879 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:07:16.563318 9797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.565068 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.564710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54528)
0806 09:07:16.564864 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.564955 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:07:16.565265 9846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
I20260806 09:07:16.577980 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:46313
--webserver_interface=127.27.187.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:16.614632 9846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 1
I20260806 09:07:16.616039 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.615557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54544)
0806 09:07:16.615774 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.615885 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 09:07:16.617219 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.616774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54546)
0806 09:07:16.616977 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.617071 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 09:07:16.617468 9846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.620977 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } attempt: 5
Aug 06 09:07:16 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:07:16.636248 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.621909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.27.187.189:44173)
0806 09:07:16.622176 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:16.622182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:16.622208 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:07:16.622335 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:16.622339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:16.622432 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:16.622551 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:16.623420 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.624154 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.625375 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.626023 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.626178 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.629838 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:16.629844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:16.629847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:16.629886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:16.633092 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.634115 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.634122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.634126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.634294 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:16.634727 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:16.634737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:16.634740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:16.635064 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:16.635241 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:16.635625 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 09:07:16.635877 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:07:16.636245 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.621786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48546)
0806 09:07:16.622005 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.622028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:16.622177 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:16.622599 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:16.622606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:16.623060 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:16.623248 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.623258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.624305 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.624311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.625215 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:16.625228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:16.626195 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:16.626201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:16.626403 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:16.627233 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:16.627260 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:16.629585 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:16.633454 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.633460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.633465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.633967 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.634424 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:16.634429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:16.634433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:16.634581 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:16.635187 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:16.635191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:16.635363 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 09:07:16.635517 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:07:16.635893 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:07:16.637526 9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:07:16.638901 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.638386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54548)
0806 09:07:16.638591 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.638751 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 09:07:16.640162 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.639728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54556)
0806 09:07:16.639894 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.640001 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:07:16.640412 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.685751 9846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 2
I20260806 09:07:16.687152 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.686557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54568)
0806 09:07:16.686827 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.686947 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:07:16.688365 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.687955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54572)
0806 09:07:16.688128 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.688218 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:07:16.688673 9846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.691668 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 1
I20260806 09:07:16.692924 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.692388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54588)
0806 09:07:16.692615 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.692723 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:07:16.694137 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.693746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54602)
0806 09:07:16.693905 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.694004 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:07:16.694407 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.762665 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 2
I20260806 09:07:16.764076 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.763451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54614)
0806 09:07:16.763728 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.763852 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:07:16.765410 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.764989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54620)
0806 09:07:16.765160 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.765259 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:07:16.765710 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.779925 9846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 3
I20260806 09:07:16.781294 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.780702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54630)
0806 09:07:16.780967 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.781083 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:07:16.782748 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.782033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54642)
0806 09:07:16.782343 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.782485 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260806 09:07:16.783025 9846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.871985 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 3
I20260806 09:07:16.873436 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.872814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54646)
0806 09:07:16.873079 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.873210 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 09:07:16.874820 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.874246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54658)
0806 09:07:16.874482 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.874637 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 09:07:16.875051 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:16.926303 9846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 4
I20260806 09:07:16.927765 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.927145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54670)
0806 09:07:16.927396 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.927517 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:07:16.928956 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:16.928531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54676)
0806 09:07:16.928714 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:16.928811 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 09:07:16.929252 9846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
W20260806 09:07:16.945816 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:16.946178 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:16.946275 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:16.954085 9854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:16.954229 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:16.954282 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:16.954334 9854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:16.954380 9854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:16.965318 9854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:35519,127.27.187.189:44173,127.27.187.188:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.187.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:16.966807 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:16.968197 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:16.981220 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:16.981109 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:16.981184 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:16.981792 9854 server_base.cc:1034] running on GCE node
I20260806 09:07:16.982846 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:16.984411 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:16.985687 9854 hybrid_clock.cc:648] HybridClock initialized: now 1786007236985638 us; error 50 us; skew 500 ppm
I20260806 09:07:16.986115 9854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:16 dist-test-slave-8tdl krb5kdc[9689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:16.989845 9854 init.cc:377] Logged in from keytab as kudu/127.27.187.188@KRBTEST.COM (short username kudu)
I20260806 09:07:16.991794 9854 webserver.cc:460] Webserver started at http://127.27.187.188:35571/ using document root <none> and password file <none>
I20260806 09:07:16.992636 9854 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:16.992798 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:16.993155 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:16.996735 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data/instance:
uuid: "989ce8ec71184490aa4a7d8c536c568f"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "895884f1616a01e0f5995c0526586497"
server_key_iv: "c3b3c1c696e6438664ac2c7464ab8b33"
server_key_version: "encryptionkey@0"
I20260806 09:07:16.997712 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal/instance:
uuid: "989ce8ec71184490aa4a7d8c536c568f"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "895884f1616a01e0f5995c0526586497"
server_key_iv: "c3b3c1c696e6438664ac2c7464ab8b33"
server_key_version: "encryptionkey@0"
I20260806 09:07:17.004367 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:07:17.008867 9869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:17.010813 9854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:07:17.011031 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "989ce8ec71184490aa4a7d8c536c568f"
format_stamp: "Formatted at 2026-08-06 09:07:16 on dist-test-slave-8tdl"
server_key: "895884f1616a01e0f5995c0526586497"
server_key_iv: "c3b3c1c696e6438664ac2c7464ab8b33"
server_key_version: "encryptionkey@0"
I20260806 09:07:17.011276 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:17.030548 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:17.035578 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:17.035879 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:17.052316 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 4
I20260806 09:07:17.053787 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.053216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54680)
0806 09:07:17.053414 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.053586 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:07:17.055197 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.054650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54690)
0806 09:07:17.054882 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.054994 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 09:07:17.055445 9771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:46313: Network error: Client connection negotiation failed: client connection to 127.27.187.188:46313: connect: Connection refused (error 111)
I20260806 09:07:17.056285 9854 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.188:46313
I20260806 09:07:17.056324 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.188:46313 every 8 connection(s)
I20260806 09:07:17.057901 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
I20260806 09:07:17.059425 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 9854
I20260806 09:07:17.059643 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007003447047-28398-0/minicluster-data/master-2/wal/instance
I20260806 09:07:17.060110 28398 external_mini_cluster.cc:1428] Setting key a372aedb4b402bcadfb3762f0c724ebd
I20260806 09:07:17.062772 9921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:17.072196 9921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:17.088428 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.062194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.27.187.188:46313)
0806 09:07:17.063129 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.063145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.063199 (+ 54us) server_negotiation.cc:407] Connection header received
0806 09:07:17.065597 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.065626 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.066187 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.066710 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.068336 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.070530 (+ 2194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.072101 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.073253 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.073452 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.079391 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.079402 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.079417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.079468 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.082976 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.083979 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.083986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.083993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.084226 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.084743 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.084748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.084752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.085588 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.085832 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.086444 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.086991 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":396,"threads_started":1}
I20260806 09:07:17.094228 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.075661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35519 (local address 127.0.0.1:59430)
0806 09:07:17.076527 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.076565 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:17.076751 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:17.077282 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:17.077289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:17.078200 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:17.078503 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.078516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.079811 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.079819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.081092 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.081106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.082028 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.082035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.082224 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.082903 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:17.082984 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:17.086038 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:17.090932 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.090943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.090948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.091678 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.092174 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.092180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.092184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.092351 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.093025 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:17.093036 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:17.093316 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 09:07:17.093573 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 09:07:17.093888 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"spinlock_wait_cycles":17280,"thread_start_us":494,"threads_started":1}
I20260806 09:07:17.094576 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.075570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.27.187.190:35519)
0806 09:07:17.076162 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.076170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.076662 (+ 492us) server_negotiation.cc:407] Connection header received
0806 09:07:17.076917 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.076921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.076994 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.077111 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.078694 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.079593 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.081317 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.081796 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.081962 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.086911 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.086917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.086920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.086959 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.090746 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.091825 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.091831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.091835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.092049 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.092502 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.092507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.092510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.092871 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.093000 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.093718 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.093986 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":278,"threads_started":1}
I20260806 09:07:17.096065 9921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:07:17.111023 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.092182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.27.187.190:35519)
0806 09:07:17.092975 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.092982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.093008 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:07:17.093062 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.093066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.093146 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.093298 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.095131 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.096212 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.098269 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.098813 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.098949 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.103925 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.103934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.103939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.103987 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.107225 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.108202 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.108207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.108210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.108476 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.109000 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.109007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.109011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.109407 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.109751 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.110276 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.110530 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":349,"threads_started":1}
I20260806 09:07:17.111521 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.096966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:44173 (local address 127.0.0.1:48560)
0806 09:07:17.097145 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.097163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:17.097250 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:17.097638 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:17.097641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:17.097981 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:17.098169 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.098179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.099245 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.099253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.100125 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.100137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.100891 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.100897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.101109 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.101675 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:17.101692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:17.104495 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:17.108401 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.108408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.108412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.109046 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.109525 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.109531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.109535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.109754 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.110527 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:17.110532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:17.110758 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 09:07:17.110952 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 09:07:17.111286 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:07:17.112154 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.097067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.27.187.189:44173)
0806 09:07:17.097302 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.097309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.097329 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:07:17.097393 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.097398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.097481 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.097596 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.098339 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.099105 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.100276 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.100723 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.100854 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.104791 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.104799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.104805 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.104855 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.108197 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.109216 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.109221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.109223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.109378 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.110000 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.110005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.110008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.110374 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.110561 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.111043 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.111877 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:07:17.113390 9921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:07:17.129410 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.114380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54718)
0806 09:07:17.114614 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.114634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:17.114786 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:17.115366 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:17.115371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:17.115913 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:17.116075 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.116085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.117050 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.117056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.117927 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.117940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.118667 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.118725 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.118922 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.119611 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:17.119629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:17.122372 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:17.126019 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.126027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.126034 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.126787 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.127276 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.127282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.127286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.127463 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.128226 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:17.128231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:17.128404 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 09:07:17.128555 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 09:07:17.128919 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 09:07:17.129491 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.114845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.27.187.188:46313)
0806 09:07:17.115013 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.115017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.115033 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:07:17.115098 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.115103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.115184 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.115297 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.116213 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.116905 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.118086 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.118504 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.118647 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.122654 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.122660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.122663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.122745 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.125820 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.126956 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.126961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.126963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.127149 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.127647 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.127652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.127679 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.128056 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.128236 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.128684 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.129109 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":56}
I20260806 09:07:17.131065 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.113449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.27.187.189:44173)
0806 09:07:17.113654 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.113661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.113748 (+ 87us) server_negotiation.cc:407] Connection header received
0806 09:07:17.113983 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.113989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.114107 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.114245 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.116021 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.116949 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.118420 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.119001 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.119154 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.124292 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.124298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.124301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.124341 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.127829 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.128618 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.128624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.128628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.128839 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.129267 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.129273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.129277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.129657 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.129989 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.130467 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.130716 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:07:17.136458 9921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: Bootstrap starting.
I20260806 09:07:17.137223 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:07:17.141494 9921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:17.143251 9921 log.cc:824] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:17.147615 9921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: No bootstrap required, opened a new log
I20260806 09:07:17.163412 9921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.164026 9921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:17.164227 9921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989ce8ec71184490aa4a7d8c536c568f, State: Initialized, Role: FOLLOWER
I20260806 09:07:17.165063 9921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.166335 9936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.167063 9936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.168165 9921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:17.171660 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.144971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.27.187.190:35519)
0806 09:07:17.145174 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.145180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.145210 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:07:17.145498 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.145504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.145602 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.145685 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.149397 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.151169 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.154450 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.155478 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.155620 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.162205 (+ 6585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.162211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.162214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.162259 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.166953 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.168079 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.168086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.168091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.168301 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.168857 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.168863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.168867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.169404 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.170785 (+ 1381us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.170954 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.171273 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:07:17.184718 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.144416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.27.187.189:44173)
0806 09:07:17.144648 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.144658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.157855 (+ 13197us) server_negotiation.cc:407] Connection header received
0806 09:07:17.158245 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.158252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.158355 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.158477 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.162184 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.163353 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.167780 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.168521 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.168645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.175420 (+ 6775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.175429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.175436 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.175492 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.179944 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.181212 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.181218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.181220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.181516 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.182127 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.182131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.182133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.182505 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.184031 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.184218 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.184440 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:07:17.188746 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.145559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.27.187.188:46313)
0806 09:07:17.145783 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.145788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.161950 (+ 16162us) server_negotiation.cc:407] Connection header received
0806 09:07:17.163017 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.163024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.163160 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.163417 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.167723 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.169083 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.171692 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.172395 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.172482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.177339 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.177348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.177355 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.177415 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.182462 (+ 5047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.183542 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.183548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.183552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.183746 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.185473 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.185478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.185481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.186077 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.188017 (+ 1940us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.188145 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.188375 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 09:07:17.192991 9947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:07:17.193176 9947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:07:17.184346 9854 master_runner.cc:186] Error getting master registration for 127.27.187.190:35519: OK, instance_id { permanent_uuid: "036438acae414985959027521ab3ab40" instance_seqno: 1786007235898539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:07:17.197324 9854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:07:17.228511 9846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 5
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:07:17.246982 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.229295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54734)
0806 09:07:17.229494 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.229515 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:17.229667 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:17.230049 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:17.230055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:17.230627 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:17.230891 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.230907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.232624 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.232630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.233899 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.233913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.235240 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.235246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.235413 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.236328 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:17.236349 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:17.239066 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:17.243549 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.243554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.243557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.244075 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.244596 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.244601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.244603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.244788 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.245642 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:17.245653 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:17.245862 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 09:07:17.246045 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 09:07:17.246502 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:07:17.248034 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.229513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.27.187.188:46313)
0806 09:07:17.229708 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.229717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.229737 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:07:17.229794 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.229797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.229881 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.229979 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.231111 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.232455 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.234067 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.234994 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.235149 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.239340 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.239349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.239354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.239401 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.243349 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.244233 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.244239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.244243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.244423 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.244925 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.244935 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.244939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.245331 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.245490 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.247356 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.247632 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:07:17.252861 9846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: Bootstrap starting.
I20260806 09:07:17.255458 9936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:07:17.255853 9936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.256796 9846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:17.257774 9936 leader_election.cc:288] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036438acae414985959027521ab3ab40 (127.27.187.190:35519), 66d70802ec5e4865bea7969f3620479b (127.27.187.189:44173)
I20260806 09:07:17.258275 9846 log.cc:824] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:17.262162 9846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: No bootstrap required, opened a new log
I20260806 09:07:17.271442 9846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.271872 9846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:17.272024 9846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d70802ec5e4865bea7969f3620479b, State: Initialized, Role: FOLLOWER
I20260806 09:07:17.272754 9846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.273859 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.274302 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.275236 9846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:17.275338 9820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "989ce8ec71184490aa4a7d8c536c568f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d70802ec5e4865bea7969f3620479b" is_pre_election: true
I20260806 09:07:17.276014 9820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 989ce8ec71184490aa4a7d8c536c568f in term 0.
I20260806 09:07:17.276867 9870 leader_election.cc:302] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d70802ec5e4865bea7969f3620479b, 989ce8ec71184490aa4a7d8c536c568f; no voters:
I20260806 09:07:17.277647 9936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:07:17.277833 9936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:07:17.277954 9936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:17.281534 9936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.283008 9819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "989ce8ec71184490aa4a7d8c536c568f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d70802ec5e4865bea7969f3620479b"
I20260806 09:07:17.283375 9819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:17.284350 9936 leader_election.cc:288] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [CANDIDATE]: Term 1 election: Requested vote from peers 036438acae414985959027521ab3ab40 (127.27.187.190:35519), 66d70802ec5e4865bea7969f3620479b (127.27.187.189:44173)
I20260806 09:07:17.287856 9819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 989ce8ec71184490aa4a7d8c536c568f in term 1.
W20260806 09:07:17.288220 9875 tablet.cc:2369] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:07:17.288470 9870 leader_election.cc:302] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d70802ec5e4865bea7969f3620479b, 989ce8ec71184490aa4a7d8c536c568f; no voters:
I20260806 09:07:17.288813 9936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:17.289856 9936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [term 1 LEADER]: Becoming Leader. State: Replica: 989ce8ec71184490aa4a7d8c536c568f, State: Running, Role: LEADER
I20260806 09:07:17.290779 9936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.296782 9956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989ce8ec71184490aa4a7d8c536c568f. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.297943 9956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:17.299603 9963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:17.302891 9963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 09:07:17.325352 9965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 09:07:17.325416 9819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [term 1 FOLLOWER]: Refusing update from remote peer 989ce8ec71184490aa4a7d8c536c568f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 09:07:17.325516 9965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:07:17.305114 9778 master_runner.cc:186] Error getting master registration for 127.27.187.190:35519: OK, instance_id { permanent_uuid: "036438acae414985959027521ab3ab40" instance_seqno: 1786007235898539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:07:17.326699 9778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:07:17.326519 9956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:17.332693 9771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } attempt: 5
Aug 06 09:07:17 dist-test-slave-8tdl krb5kdc[9689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:07:17.353349 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.333420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:46313 (local address 127.0.0.1:54748)
0806 09:07:17.333869 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:17.333892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:17.334012 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:17.334377 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:17.334382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:17.334969 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:17.335207 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.335222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.336881 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.336887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.338496 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:17.338514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.340234 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.340244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:17.340482 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.341415 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:17.341455 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:17.344597 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:17.349862 (+ 5265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.349867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.349872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.350493 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.351131 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:17.351137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:17.351143 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:17.351333 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:17.352142 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:17.352150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:17.352349 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:07:17.352515 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 09:07:17.352853 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260806 09:07:17.354388 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:17.333844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.27.187.188:46313)
0806 09:07:17.334018 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:17.334025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:17.334045 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:07:17.334130 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:17.334138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:17.334220 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:17.334323 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:17.335404 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.336580 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.338959 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:17.339767 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:17.339897 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:17.344856 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:17.344863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:17.344869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:17.344919 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:17.349580 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.350716 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.350722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.350726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.350955 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:17.351471 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:17.351477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:17.351480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:17.351885 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:17.352060 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:17.352831 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 09:07:17.353894 (+ 1063us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 09:07:17.359725 9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: Bootstrap starting.
I20260806 09:07:17.359855 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989ce8ec71184490aa4a7d8c536c568f. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.360229 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.365839 9967 mvcc.cc:205] Tried to move back new op lower bound from 7315485644072693760 to 7315485643948453888. Current Snapshot: MvccSnapshot[applied={T|T < 7315485644072693760}]
I20260806 09:07:17.366932 9771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:17.368636 9771 log.cc:824] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:17.369378 9936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.369781 9936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:17.370369 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.370723 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.372623 9963 catalog_manager.cc:1347] Generated new cluster ID: 87c1254b3f3a49429b68e52421c48189
I20260806 09:07:17.372733 9963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:17.393174 9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: No bootstrap required, opened a new log
I20260806 09:07:17.415807 9963 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:07:17.418090 9963 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:17.419565 9771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.420706 9771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:17.420996 9771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036438acae414985959027521ab3ab40, State: Initialized, Role: FOLLOWER
I20260806 09:07:17.422631 9771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } }
I20260806 09:07:17.438303 9744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:17.438879 9745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "989ce8ec71184490aa4a7d8c536c568f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036438acae414985959027521ab3ab40"
I20260806 09:07:17.440618 9746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "989ce8ec71184490aa4a7d8c536c568f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036438acae414985959027521ab3ab40" is_pre_election: true
I20260806 09:07:17.442368 9963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f: Generated new TSK 0
I20260806 09:07:17.443637 9963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:07:17.445746 9744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [term 1 FOLLOWER]: Refusing update from remote peer 989ce8ec71184490aa4a7d8c536c568f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:17.446738 9936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [LEADER]: Connected to new peer: Peer: permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:17.447082 9771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:17.446705 9973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.449774 9973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.483690 9977 mvcc.cc:205] Tried to move back new op lower bound from 7315485644072693760 to 7315485643948453888. Current Snapshot: MvccSnapshot[applied={T|T < 7315485644072693760}]
I20260806 09:07:17.494637 9973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989ce8ec71184490aa4a7d8c536c568f. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.495046 9973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.495623 9974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.496088 9974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:07:17.593595 9993 catalog_manager.cc:1259] Loaded cluster ID: 87c1254b3f3a49429b68e52421c48189
I20260806 09:07:17.593744 9993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: loading cluster ID for follower catalog manager: success
I20260806 09:07:17.598402 9993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: acquiring CA information for follower catalog manager: success
I20260806 09:07:17.601524 9993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 036438acae414985959027521ab3ab40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:07:17.842310 9936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "989ce8ec71184490aa4a7d8c536c568f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "036438acae414985959027521ab3ab40" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35519 } } peers { permanent_uuid: "66d70802ec5e4865bea7969f3620479b" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 44173 } } peers { permanent_uuid: "989ce8ec71184490aa4a7d8c536c568f" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 46313 } } }
I20260806 09:07:17.842670 9936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989ce8ec71184490aa4a7d8c536c568f [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:18.328352 9965 catalog_manager.cc:1259] Loaded cluster ID: 87c1254b3f3a49429b68e52421c48189
I20260806 09:07:18.328470 9965 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: loading cluster ID for follower catalog manager: success
I20260806 09:07:18.331293 9965 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: acquiring CA information for follower catalog manager: success
I20260806 09:07:18.333369 9965 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66d70802ec5e4865bea7969f3620479b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:07:18.605944 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9703
I20260806 09:07:18.607908 9766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:18.720969 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9703
I20260806 09:07:18.745975 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9778
I20260806 09:07:18.747315 9841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:18.863059 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9778
I20260806 09:07:18.884382 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 9854
I20260806 09:07:18.885911 9916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:19.004199 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 9854
2026-08-06T09:07:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5115 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:07:19 dist-test-slave-8tdl krb5kdc[10002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:07:19 dist-test-slave-8tdl krb5kdc[10002](info): set up 2 sockets
Aug 06 09:07:19 dist-test-slave-8tdl krb5kdc[10002](info): commencing operation
krb5kdc: starting...
W20260806 09:07:20.191071 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:07:20 dist-test-slave-8tdl krb5kdc[10002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:07:20.523769 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43751
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:20.904752 10015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:20.905086 10015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:20.905261 10015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:20.913156 10015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:20.913311 10015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:20.913411 10015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:20.913508 10015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:20.913617 10015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:20.924615 10015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:20.926136 10015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:20.927603 10015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:20.941243 10020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:20.941114 10021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:20.941373 10023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:20.942575 10015 server_base.cc:1034] running on GCE node
I20260806 09:07:20.943620 10015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:20.945405 10015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:20.946740 10015 hybrid_clock.cc:648] HybridClock initialized: now 1786007240946598 us; error 81 us; skew 500 ppm
I20260806 09:07:20.947273 10015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:20 dist-test-slave-8tdl krb5kdc[10002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007240, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:20.954983 10015 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:20.956923 10015 webserver.cc:460] Webserver started at http://127.27.187.190:32809/ using document root <none> and password file <none>
I20260806 09:07:20.957736 10015 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:20.957908 10015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:20.958261 10015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:20.961735 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "e76cfead43664c759980505ee597f9c9"
format_stamp: "Formatted at 2026-08-06 09:07:20 on dist-test-slave-8tdl"
server_key: "b3ce7cf3fc737d80b1f26cb958615ec1"
server_key_iv: "aa9f268efffa064e3acdde31371f11fe"
server_key_version: "encryptionkey@0"
I20260806 09:07:20.962634 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "e76cfead43664c759980505ee597f9c9"
format_stamp: "Formatted at 2026-08-06 09:07:20 on dist-test-slave-8tdl"
server_key: "b3ce7cf3fc737d80b1f26cb958615ec1"
server_key_iv: "aa9f268efffa064e3acdde31371f11fe"
server_key_version: "encryptionkey@0"
I20260806 09:07:20.969184 10015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:07:20.974058 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:20.976166 10015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:07:20.976430 10015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "e76cfead43664c759980505ee597f9c9"
format_stamp: "Formatted at 2026-08-06 09:07:20 on dist-test-slave-8tdl"
server_key: "b3ce7cf3fc737d80b1f26cb958615ec1"
server_key_iv: "aa9f268efffa064e3acdde31371f11fe"
server_key_version: "encryptionkey@0"
I20260806 09:07:20.976684 10015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:20.996861 10015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:21.002149 10015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:21.002494 10015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:21.026444 10015 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:43751
I20260806 09:07:21.026468 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:43751 every 8 connection(s)
I20260806 09:07:21.028183 10015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:21.032521 10083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:21.037912 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10015
I20260806 09:07:21.038275 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:07:21.039276 28398 external_mini_cluster.cc:1428] Setting key 99e456d9d65957aa9bd84693724b74eb
I20260806 09:07:21.045047 10083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: Bootstrap starting.
I20260806 09:07:21.049322 10083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:21.050858 10083 log.cc:824] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:21.055299 10083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: No bootstrap required, opened a new log
Aug 06 09:07:21 dist-test-slave-8tdl krb5kdc[10002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:21.064088 10083 raft_consensus.cc:348] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } }
I20260806 09:07:21.064586 10083 raft_consensus.cc:374] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:21.064724 10083 raft_consensus.cc:729] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76cfead43664c759980505ee597f9c9, State: Initialized, Role: FOLLOWER
I20260806 09:07:21.065448 10083 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } }
I20260806 09:07:21.065737 10083 raft_consensus.cc:388] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:21.065922 10083 raft_consensus.cc:482] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:21.066145 10083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:21.067436 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:21.041366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.27.187.190:43751)
0806 09:07:21.042298 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:21.042311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:21.042359 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:07:21.044296 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:21.044324 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:21.044810 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:21.045182 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:21.047446 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:21.049177 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:21.051287 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:21.052603 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:21.052795 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:21.058408 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:21.058417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:21.058441 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:21.058497 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:21.062217 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:21.063250 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:21.063259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:21.063267 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:21.063473 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:21.063959 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:21.063963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:21.063965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:21.064685 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:21.065041 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:21.065579 (+ 538us) server_negotiation.cc:299] Negotiation successful
0806 09:07:21.066124 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":418,"threads_started":1}
I20260806 09:07:21.069386 10083 raft_consensus.cc:504] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } }
I20260806 09:07:21.069942 10083 leader_election.cc:302] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e76cfead43664c759980505ee597f9c9; no voters:
I20260806 09:07:21.071256 10083 leader_election.cc:288] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:07:21.071426 10088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:21.072605 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:07:21.073544 10083 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:21.077356 10088 raft_consensus.cc:686] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [term 1 LEADER]: Becoming Leader. State: Replica: e76cfead43664c759980505ee597f9c9, State: Running, Role: LEADER
I20260806 09:07:21.078663 10088 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } }
I20260806 09:07:21.084239 10089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e76cfead43664c759980505ee597f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } } }
I20260806 09:07:21.084359 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76cfead43664c759980505ee597f9c9. Latest consensus state: current_term: 1 leader_uuid: "e76cfead43664c759980505ee597f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76cfead43664c759980505ee597f9c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43751 } } }
I20260806 09:07:21.084838 10089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:21.084839 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9 [sys.catalog]: This master's current role is: LEADER
W20260806 09:07:21.090992 10102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:07:21.091137 10102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:07:21.091315 10103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:21.094161 10103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:21.104624 10103 catalog_manager.cc:1347] Generated new cluster ID: 1d7c2504075c4bcca53c8c8b38b9fb23
I20260806 09:07:21.104777 10103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:21.121122 10103 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:07:21.123229 10103 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:21.145169 10103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e76cfead43664c759980505ee597f9c9: Generated new TSK 0
I20260806 09:07:21.146440 10103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:07:21.192052 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10015
I20260806 09:07:21.193629 10078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:21.300800 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10015
2026-08-06T09:07:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2294 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:07:21 dist-test-slave-8tdl krb5kdc[10111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:07:21 dist-test-slave-8tdl krb5kdc[10111](info): set up 2 sockets
Aug 06 09:07:21 dist-test-slave-8tdl krb5kdc[10111](info): commencing operation
krb5kdc: starting...
W20260806 09:07:22.466598 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.008s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:07:22 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:07:22.789023 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:23.165360 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:23.165710 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:23.165900 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:23.173489 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:23.173623 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:23.173710 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:23.173828 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:23.173914 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:23.184749 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:23.186272 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:23.187743 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:23.201336 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:23.201223 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:23.201243 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:23.202183 10124 server_base.cc:1034] running on GCE node
I20260806 09:07:23.203197 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:23.204838 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:23.206127 10124 hybrid_clock.cc:648] HybridClock initialized: now 1786007243206047 us; error 69 us; skew 500 ppm
I20260806 09:07:23.206588 10124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:23 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:23.210553 10124 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:23.212543 10124 webserver.cc:460] Webserver started at http://127.27.187.190:39801/ using document root <none> and password file <none>
I20260806 09:07:23.213327 10124 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:23.213450 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:23.213755 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:23.217459 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "600a390d245442308c3fbc273182ae26"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "8ccbae1947c92e65a82dba21f8c86565"
server_key_iv: "294d6330ee62673f7e9118750505a1b8"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.218355 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "600a390d245442308c3fbc273182ae26"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "8ccbae1947c92e65a82dba21f8c86565"
server_key_iv: "294d6330ee62673f7e9118750505a1b8"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.224694 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:07:23.229213 10139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:23.231166 10124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 09:07:23.231396 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "600a390d245442308c3fbc273182ae26"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "8ccbae1947c92e65a82dba21f8c86565"
server_key_iv: "294d6330ee62673f7e9118750505a1b8"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.231659 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:23.273641 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:23.278825 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:23.279175 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:23.300930 10124 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34611
I20260806 09:07:23.300966 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34611 every 8 connection(s)
I20260806 09:07:23.302477 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:23.303773 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10124
I20260806 09:07:23.304041 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:07:23.304726 28398 external_mini_cluster.cc:1428] Setting key a6e184336de3044f8207900bd2e24f4f
I20260806 09:07:23.307168 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:07:23 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:23.321676 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap starting.
I20260806 09:07:23.325735 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:23.327055 10191 log.cc:824] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:23.331003 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: No bootstrap required, opened a new log
I20260806 09:07:23.332396 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:23.306883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.27.187.190:34611)
0806 09:07:23.307751 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:23.307767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:23.307822 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:07:23.310234 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:23.310266 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:23.310867 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:23.311386 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:23.312924 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:23.315120 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:23.316655 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:23.317818 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:23.318022 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:23.322989 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:23.323000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:23.323019 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:23.323144 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:23.327003 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:23.328146 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:23.328158 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:23.328169 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:23.328428 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:23.328969 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:23.328974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:23.328976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:23.329620 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:23.329898 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:23.330527 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 09:07:23.331211 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":343,"threads_started":1}
I20260806 09:07:23.339933 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:23.340366 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:23.340498 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600a390d245442308c3fbc273182ae26, State: Initialized, Role: FOLLOWER
I20260806 09:07:23.341157 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:23.341441 10191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:23.341603 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:23.341775 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:23.344302 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:23.344884 10191 leader_election.cc:302] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600a390d245442308c3fbc273182ae26; no voters:
I20260806 09:07:23.345584 10191 leader_election.cc:288] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:07:23.345763 10196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:23.347038 10196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 LEADER]: Becoming Leader. State: Replica: 600a390d245442308c3fbc273182ae26, State: Running, Role: LEADER
I20260806 09:07:23.347695 10196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:23.348111 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:23.350880 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:23.350870 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600a390d245442308c3fbc273182ae26. Latest consensus state: current_term: 1 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:23.353709 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:23.353782 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
W20260806 09:07:23.361193 10211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:07:23.361331 10211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:07:23.361493 10209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:23.363950 10209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:23.373836 10209 catalog_manager.cc:1347] Generated new cluster ID: 9de39faa14824caebe97c5439514247f
I20260806 09:07:23.373978 10209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:23.385255 10209 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:07:23.386768 10209 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:23.393774 10209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Generated new TSK 0
I20260806 09:07:23.394937 10209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:07:23.476038 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:23.862445 10218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:23.862833 10218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:23.862990 10218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:23.870754 10218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:23.870913 10218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:23.871142 10218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:23.881737 10218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:23.883322 10218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:23.884806 10218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:23.901018 10223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:23.901186 10224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:23.901507 10226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:23.902616 10218 server_base.cc:1034] running on GCE node
I20260806 09:07:23.903481 10218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:23.904613 10218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:23.905884 10218 hybrid_clock.cc:648] HybridClock initialized: now 1786007243905815 us; error 61 us; skew 500 ppm
I20260806 09:07:23.906261 10218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:23 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:23.910053 10218 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:23.911975 10218 webserver.cc:460] Webserver started at http://127.27.187.129:40559/ using document root <none> and password file <none>
I20260806 09:07:23.912914 10218 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:23.913096 10218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:23.913462 10218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:23.917331 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "666c40aa069c417b88c8add306391578"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "c80518564d7070e524cddb487947079b"
server_key_iv: "b61c71c8ad0f151e2a37812c479a2479"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.918519 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "666c40aa069c417b88c8add306391578"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "c80518564d7070e524cddb487947079b"
server_key_iv: "b61c71c8ad0f151e2a37812c479a2479"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.925261 10218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:07:23.929620 10233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:23.931438 10218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:07:23.931697 10218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "666c40aa069c417b88c8add306391578"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "c80518564d7070e524cddb487947079b"
server_key_iv: "b61c71c8ad0f151e2a37812c479a2479"
server_key_version: "encryptionkey@0"
I20260806 09:07:23.931963 10218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:23.970073 10218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:23.974962 10218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:23.975324 10218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:23.976434 10218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:23.977972 10218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:23.978084 10218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:23.978443 10218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:23.978541 10218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:24.019840 10218 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:38595
I20260806 09:07:24.019878 10346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:38595 every 8 connection(s)
I20260806 09:07:24.021381 10218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:24.029585 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10218
I20260806 09:07:24.029925 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:07:24.030759 28398 external_mini_cluster.cc:1428] Setting key e22f327c675a5acf0ee7f162536d2db1
Aug 06 09:07:24 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:24.045331 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:24.024840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:34441)
0806 09:07:24.025682 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:24.025733 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:24.027641 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:24.028457 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:24.028465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:24.029026 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:24.029640 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:24.029661 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.030882 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.030887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:24.032385 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:24.032399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.033317 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.033327 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:24.033567 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:24.034380 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:24.034410 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:24.037473 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:24.040932 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:24.040944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:24.040965 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:24.041488 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:24.041910 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:24.041915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:24.041919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:24.042131 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:24.042738 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:24.042746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:24.043135 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 09:07:24.043538 (+ 403us) client_negotiation.cc:240] Negotiation successful
0806 09:07:24.044044 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":333,"threads_started":1}
I20260806 09:07:24.045894 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:24.024330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:34441 (local address 127.27.187.190:34611)
0806 09:07:24.024856 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:24.024861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:24.025873 (+ 1012us) server_negotiation.cc:407] Connection header received
0806 09:07:24.027947 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:24.027953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:24.028068 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:24.028201 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:24.029854 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.030663 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.032544 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.033038 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.033184 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:24.037790 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:24.037796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:24.037800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:24.037839 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:24.040750 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:24.041631 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:24.041637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:24.041640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:24.041787 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:24.042250 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:24.042257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:24.042260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:24.042561 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:24.042763 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:24.045199 (+ 2436us) server_negotiation.cc:299] Negotiation successful
0806 09:07:24.045449 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
I20260806 09:07:24.047546 10347 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:24.048003 10347 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:24.049006 10347 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:24.051841 10156 ts_manager.cc:194] Registered new tserver with Master: 666c40aa069c417b88c8add306391578 (127.27.187.129:38595)
I20260806 09:07:24.054541 10156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:34441
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:07:24.107218 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:24.485024 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:24.485404 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:24.485596 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:24.493491 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:24.493680 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:24.493883 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:24.504597 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:24.506099 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:24.507608 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:24.522409 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:24.522363 10362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:24.522457 10360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:24.523394 10353 server_base.cc:1034] running on GCE node
I20260806 09:07:24.524121 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:24.525348 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:24.526607 10353 hybrid_clock.cc:648] HybridClock initialized: now 1786007244526549 us; error 36 us; skew 500 ppm
I20260806 09:07:24.527134 10353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:24 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007244, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:24.531247 10353 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:24.533203 10353 webserver.cc:460] Webserver started at http://127.27.187.130:38369/ using document root <none> and password file <none>
I20260806 09:07:24.534183 10353 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:24.534343 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:24.534770 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:24.538726 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "ef604b59e0814ec683979c83bfcffc8f"
format_stamp: "Formatted at 2026-08-06 09:07:24 on dist-test-slave-8tdl"
server_key: "3c8792c1d6549e445303cfab3655fe05"
server_key_iv: "55fa6c1b5467e7710da1fd67cea351d5"
server_key_version: "encryptionkey@0"
I20260806 09:07:24.539736 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "ef604b59e0814ec683979c83bfcffc8f"
format_stamp: "Formatted at 2026-08-06 09:07:24 on dist-test-slave-8tdl"
server_key: "3c8792c1d6549e445303cfab3655fe05"
server_key_iv: "55fa6c1b5467e7710da1fd67cea351d5"
server_key_version: "encryptionkey@0"
I20260806 09:07:24.546386 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:07:24.551009 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:24.553112 10353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:07:24.553350 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ef604b59e0814ec683979c83bfcffc8f"
format_stamp: "Formatted at 2026-08-06 09:07:24 on dist-test-slave-8tdl"
server_key: "3c8792c1d6549e445303cfab3655fe05"
server_key_iv: "55fa6c1b5467e7710da1fd67cea351d5"
server_key_version: "encryptionkey@0"
I20260806 09:07:24.553586 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:24.587121 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:24.592465 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:24.592842 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:24.593976 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:24.595546 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:24.595624 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:24.595816 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:24.595885 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:24.633983 10353 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36163
I20260806 09:07:24.634088 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36163 every 8 connection(s)
I20260806 09:07:24.635571 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:07:24.640908 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10353
I20260806 09:07:24.641188 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:07:24.641716 28398 external_mini_cluster.cc:1428] Setting key 16adb8ebfc7eb46e7929e5811c7fd42f
Aug 06 09:07:24 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007244, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:24.658625 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:24.638667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:59101 (local address 127.27.187.190:34611)
0806 09:07:24.639241 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:24.639246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:24.640161 (+ 915us) server_negotiation.cc:407] Connection header received
0806 09:07:24.642347 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:24.642352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:24.642452 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:24.642591 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:24.644239 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.645109 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.646841 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.647299 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.647452 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:24.651974 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:24.651980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:24.651983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:24.652023 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:24.655116 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:24.656017 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:24.656022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:24.656025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:24.656166 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:24.656671 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:24.656681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:24.656684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:24.656995 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:24.657165 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:24.657942 (+ 777us) server_negotiation.cc:299] Negotiation successful
0806 09:07:24.658206 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260806 09:07:24.659713 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:24.639193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:59101)
0806 09:07:24.639991 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:24.640030 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:24.642028 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:24.642889 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:24.642898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:24.643456 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:24.644059 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:24.644077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.645281 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.645287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:24.646593 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:24.646606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:24.647566 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:24.647574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:24.647876 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:24.648727 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:24.648752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:24.651645 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:24.655322 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:24.655333 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:24.655355 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:24.655860 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:24.656298 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:24.656304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:24.656308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:24.656538 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:24.657138 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:24.657146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:24.657548 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 09:07:24.657840 (+ 292us) client_negotiation.cc:240] Negotiation successful
0806 09:07:24.658460 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":291,"threads_started":1}
I20260806 09:07:24.661382 10484 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:24.661720 10484 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:24.662505 10484 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:24.664291 10156 ts_manager.cc:194] Registered new tserver with Master: ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163)
I20260806 09:07:24.665807 10156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:59101
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:07:24.716030 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:25.058470 10347 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
W20260806 09:07:25.093918 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:25.094264 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:25.094370 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:25.102600 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:25.102828 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:25.102957 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:25.113706 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:25.115222 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:25.116621 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:25.132489 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:25.132601 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:25.132757 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:25.134856 10490 server_base.cc:1034] running on GCE node
I20260806 09:07:25.135833 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:25.137239 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:25.138521 10490 hybrid_clock.cc:648] HybridClock initialized: now 1786007245138423 us; error 98 us; skew 500 ppm
I20260806 09:07:25.138932 10490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:25 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007245, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:25.142864 10490 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:25.144695 10490 webserver.cc:460] Webserver started at http://127.27.187.131:36045/ using document root <none> and password file <none>
I20260806 09:07:25.145565 10490 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:25.145757 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:25.146277 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:25.150928 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
format_stamp: "Formatted at 2026-08-06 09:07:25 on dist-test-slave-8tdl"
server_key: "cc885f9897008cb57f33d50cc9f558b2"
server_key_iv: "f3d92c7079220b4d010b2bf037680082"
server_key_version: "encryptionkey@0"
I20260806 09:07:25.152536 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
format_stamp: "Formatted at 2026-08-06 09:07:25 on dist-test-slave-8tdl"
server_key: "cc885f9897008cb57f33d50cc9f558b2"
server_key_iv: "f3d92c7079220b4d010b2bf037680082"
server_key_version: "encryptionkey@0"
I20260806 09:07:25.162120 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s
I20260806 09:07:25.168640 10505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:25.170979 10490 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:07:25.171226 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
format_stamp: "Formatted at 2026-08-06 09:07:25 on dist-test-slave-8tdl"
server_key: "cc885f9897008cb57f33d50cc9f558b2"
server_key_iv: "f3d92c7079220b4d010b2bf037680082"
server_key_version: "encryptionkey@0"
I20260806 09:07:25.171559 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:25.206794 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:25.212466 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:25.212744 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:25.213800 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:25.215451 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:25.215534 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:25.215637 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:25.215685 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:25.259488 10490 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33963
I20260806 09:07:25.259704 10619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33963 every 8 connection(s)
I20260806 09:07:25.260974 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:25.270304 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10490
I20260806 09:07:25.270531 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:07:25.271310 28398 external_mini_cluster.cc:1428] Setting key e6a275b2bd2aa69f5519ff26e3df7298
Aug 06 09:07:25 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007245, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:25.285866 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.264205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:35021 (local address 127.27.187.190:34611)
0806 09:07:25.264783 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.264788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.265705 (+ 917us) server_negotiation.cc:407] Connection header received
0806 09:07:25.267949 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.267956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.268072 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.268220 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:25.270157 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.271291 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.273478 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.274038 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.274185 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.278876 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:25.278883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:25.278887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:25.278930 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:25.282189 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.283246 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.283251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.283254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.283417 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.283929 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.283933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.283936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.284268 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:25.284416 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.285250 (+ 834us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.285456 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
I20260806 09:07:25.287254 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.264551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:35021)
0806 09:07:25.265433 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.265481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.267587 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.268533 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.268543 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.269207 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:25.269918 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.269941 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.271494 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.271506 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.273301 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.273318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.274307 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.274313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.274538 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.275609 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:25.275639 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:25.278458 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:25.282434 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:25.282442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:25.282459 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:25.283089 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:25.283538 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:25.283541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:25.283543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:25.283765 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:25.284429 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:25.284437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:25.284859 (+ 422us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.285120 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.285674 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":366,"threads_started":1}
I20260806 09:07:25.289387 10620 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:25.289806 10620 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:25.290781 10620 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:25.292624 10156 ts_manager.cc:194] Registered new tserver with Master: fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963)
I20260806 09:07:25.293581 10156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:35021
I20260806 09:07:25.302361 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:07:25.324360 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.308163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.27.187.190:34611)
0806 09:07:25.308402 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.308409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.308779 (+ 370us) server_negotiation.cc:407] Connection header received
0806 09:07:25.308938 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.308942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.309026 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.309115 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:25.310632 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.311483 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.312977 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.313448 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.313553 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.316773 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:25.316780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:25.316784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:25.316828 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:25.320687 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.321749 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.321754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.321757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.321921 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.322483 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.322488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.322495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.323060 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:25.323297 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.323720 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.323959 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:07:25.329739 10156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:07:25.338274 10156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:07:25.366777 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.354158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.27.187.129:38595)
0806 09:07:25.355493 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.355507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.356657 (+ 1150us) server_negotiation.cc:407] Connection header received
0806 09:07:25.356914 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.356921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.357207 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.357399 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:25.358979 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.361004 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.363962 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.365158 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.365400 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.365637 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.366153 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.366366 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":952,"threads_started":1}
I20260806 09:07:25.367297 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.353889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46678)
0806 09:07:25.356376 (+ 2487us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.356416 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.356642 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.357705 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.357713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.357753 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:25.358645 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.358748 (+ 103us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.361156 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.361160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.363756 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.363773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.365415 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.365421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.365801 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.365879 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.366075 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.366331 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2275,"mutex_wait_us":225,"spinlock_wait_cycles":14592,"thread_start_us":2047,"threads_started":1}
I20260806 09:07:25.368219 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.356892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36163 (local address 127.0.0.1:46898)
0806 09:07:25.357455 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.357476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.357606 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.358870 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.358874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.358898 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:25.359374 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.359392 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.361077 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.361084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.364742 (+ 3658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.364761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.367153 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.367161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.367441 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.367487 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.367599 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.367774 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":403,"thread_start_us":907,"threads_started":1}
I20260806 09:07:25.369189 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.357530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.27.187.130:36163)
0806 09:07:25.358217 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.358227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.358256 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:07:25.358352 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.358359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.358628 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.359599 (+ 971us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:25.359817 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.360911 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.364966 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.366516 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.366729 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.366927 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.368564 (+ 1637us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.368784 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":1930,"threads_started":1}
I20260806 09:07:25.369439 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.354084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.27.187.131:33963)
0806 09:07:25.355821 (+ 1737us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.355827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.356568 (+ 741us) server_negotiation.cc:407] Connection header received
0806 09:07:25.356900 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.356909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.357189 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.357399 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:25.359035 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.361033 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.367045 (+ 6012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.368250 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.368499 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.368685 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.368883 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.369065 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":1439,"threads_started":1}
I20260806 09:07:25.372213 10281 tablet_service.cc:1467] Processing CreateTablet for tablet c824b227d81f41049b6d377686c9c76d (DEFAULT_TABLE table=test-table [id=be3d49392ac54b4fb6bb3e6de5dc40ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:25.373581 10281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c824b227d81f41049b6d377686c9c76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:25.373368 10417 tablet_service.cc:1467] Processing CreateTablet for tablet c824b227d81f41049b6d377686c9c76d (DEFAULT_TABLE table=test-table [id=be3d49392ac54b4fb6bb3e6de5dc40ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:25.374835 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c824b227d81f41049b6d377686c9c76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:25.376780 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.353848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33963 (local address 127.0.0.1:33610)
0806 09:07:25.356378 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.356415 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.356641 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.357715 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.357719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.357755 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:25.358635 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.358654 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.361286 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.361296 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.364742 (+ 3446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.364761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.368494 (+ 3733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.368500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.368670 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.368704 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.368821 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.376268 (+ 7447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257,"mutex_wait_us":348,"spinlock_wait_cycles":36096,"thread_start_us":4115,"threads_started":1}
I20260806 09:07:25.381932 10554 tablet_service.cc:1467] Processing CreateTablet for tablet c824b227d81f41049b6d377686c9c76d (DEFAULT_TABLE table=test-table [id=be3d49392ac54b4fb6bb3e6de5dc40ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:25.383831 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c824b227d81f41049b6d377686c9c76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:25.388443 10639 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Bootstrap starting.
I20260806 09:07:25.393072 10639 tablet_bootstrap.cc:654] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:25.394373 10639 log.cc:824] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:25.394981 10640 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Bootstrap starting.
I20260806 09:07:25.399196 10640 tablet_bootstrap.cc:654] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:25.400647 10640 log.cc:824] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:25.403126 10643 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Bootstrap starting.
I20260806 09:07:25.403321 10639 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: No bootstrap required, opened a new log
I20260806 09:07:25.403719 10639 ts_tablet_manager.cc:1397] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260806 09:07:25.404327 10640 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: No bootstrap required, opened a new log
I20260806 09:07:25.404632 10640 ts_tablet_manager.cc:1397] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 09:07:25.407804 10643 tablet_bootstrap.cc:654] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:25.409474 10643 log.cc:824] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:25.414067 10643 tablet_bootstrap.cc:492] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: No bootstrap required, opened a new log
I20260806 09:07:25.414445 10643 ts_tablet_manager.cc:1397] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260806 09:07:25.415058 10640 raft_consensus.cc:348] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.415634 10640 raft_consensus.cc:374] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:25.415755 10640 raft_consensus.cc:729] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666c40aa069c417b88c8add306391578, State: Initialized, Role: FOLLOWER
I20260806 09:07:25.415915 10639 raft_consensus.cc:348] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.416507 10639 raft_consensus.cc:374] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:25.416658 10639 raft_consensus.cc:729] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef604b59e0814ec683979c83bfcffc8f, State: Initialized, Role: FOLLOWER
I20260806 09:07:25.416486 10640 consensus_queue.cc:260] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.417411 10639 consensus_queue.cc:260] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.418951 10484 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
I20260806 09:07:25.419546 10639 ts_tablet_manager.cc:1428] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 09:07:25.422079 10640 ts_tablet_manager.cc:1428] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260806 09:07:25.426733 10643 raft_consensus.cc:348] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.427413 10643 raft_consensus.cc:374] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:25.427546 10643 raft_consensus.cc:729] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b192e40ad4544a09ec8e5b481bf43, State: Initialized, Role: FOLLOWER
I20260806 09:07:25.428211 10643 consensus_queue.cc:260] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.429477 10620 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
I20260806 09:07:25.429970 10643 ts_tablet_manager.cc:1428] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260806 09:07:25.460268 10645 raft_consensus.cc:482] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:07:25.460646 10645 raft_consensus.cc:504] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.463280 10645 leader_election.cc:288] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963), ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163)
I20260806 09:07:25.471851 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.463919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:52293 (local address 127.27.187.130:36163)
0806 09:07:25.464177 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.464187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.465361 (+ 1174us) server_negotiation.cc:407] Connection header received
0806 09:07:25.465638 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.465643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.465711 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.465828 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:25.466764 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.467514 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.469687 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.470530 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.470787 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.470883 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.471219 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.471534 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:07:25.472026 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.463947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36163 (local address 127.27.187.129:52293)
0806 09:07:25.465220 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.465236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.465420 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.465848 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.465852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.465950 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:25.466529 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.466543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.467674 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.467679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.469476 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.469497 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.470837 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.470844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.471022 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.471053 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.471182 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.471487 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":107,"spinlock_wait_cycles":29440,"thread_start_us":641,"threads_started":1}
I20260806 09:07:25.472054 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.463795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:49149 (local address 127.27.187.131:33963)
0806 09:07:25.464061 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.464070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.465203 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 09:07:25.465641 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.465646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.465712 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.465829 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:25.466582 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.467457 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.469752 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.470612 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.470909 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.470994 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.471554 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.471739 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:07:25.472785 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.463683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33963 (local address 127.27.187.129:49149)
0806 09:07:25.465064 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.465081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:25.465386 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:25.465847 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:25.465852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:25.465878 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:25.466399 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.466414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.467628 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.467634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.469506 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:25.469521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.470956 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.470963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:25.471134 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.471281 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 09:07:25.472273 (+ 992us) client_negotiation.cc:240] Negotiation successful
0806 09:07:25.472451 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":144,"thread_start_us":739,"threads_started":1}
I20260806 09:07:25.473469 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c824b227d81f41049b6d377686c9c76d" candidate_uuid: "666c40aa069c417b88c8add306391578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef604b59e0814ec683979c83bfcffc8f" is_pre_election: true
I20260806 09:07:25.474108 10438 raft_consensus.cc:2454] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666c40aa069c417b88c8add306391578 in term 0.
I20260806 09:07:25.474073 10574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c824b227d81f41049b6d377686c9c76d" candidate_uuid: "666c40aa069c417b88c8add306391578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" is_pre_election: true
I20260806 09:07:25.474741 10574 raft_consensus.cc:2454] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666c40aa069c417b88c8add306391578 in term 0.
I20260806 09:07:25.475052 10237 leader_election.cc:302] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666c40aa069c417b88c8add306391578, ef604b59e0814ec683979c83bfcffc8f; no voters:
I20260806 09:07:25.475627 10645 raft_consensus.cc:2781] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:07:25.475857 10645 raft_consensus.cc:482] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:07:25.475944 10645 raft_consensus.cc:3037] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:25.478814 10645 raft_consensus.cc:504] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.479866 10645 leader_election.cc:288] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [CANDIDATE]: Term 1 election: Requested vote from peers fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963), ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163)
I20260806 09:07:25.480641 10574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c824b227d81f41049b6d377686c9c76d" candidate_uuid: "666c40aa069c417b88c8add306391578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
I20260806 09:07:25.480690 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c824b227d81f41049b6d377686c9c76d" candidate_uuid: "666c40aa069c417b88c8add306391578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef604b59e0814ec683979c83bfcffc8f"
I20260806 09:07:25.480962 10574 raft_consensus.cc:3037] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:25.480980 10438 raft_consensus.cc:3037] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:25.483810 10574 raft_consensus.cc:2454] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666c40aa069c417b88c8add306391578 in term 1.
I20260806 09:07:25.483867 10438 raft_consensus.cc:2454] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666c40aa069c417b88c8add306391578 in term 1.
I20260806 09:07:25.484602 10235 leader_election.cc:302] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666c40aa069c417b88c8add306391578, fd8b192e40ad4544a09ec8e5b481bf43; no voters:
I20260806 09:07:25.485074 10645 raft_consensus.cc:2781] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:25.485942 10645 raft_consensus.cc:686] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 1 LEADER]: Becoming Leader. State: Replica: 666c40aa069c417b88c8add306391578, State: Running, Role: LEADER
I20260806 09:07:25.486573 10645 consensus_queue.cc:237] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } }
I20260806 09:07:25.492156 10155 catalog_manager.cc:5495] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666c40aa069c417b88c8add306391578 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "666c40aa069c417b88c8add306391578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } health_report { overall_health: UNKNOWN } } }
W20260806 09:07:25.513074 10621 tablet.cc:2369] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:07:25.524398 10348 tablet.cc:2369] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:07:25.532255 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.525480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.27.187.129:38595)
0806 09:07:25.525725 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.525730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.525821 (+ 91us) server_negotiation.cc:407] Connection header received
0806 09:07:25.526031 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.526035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.526117 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.526253 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:07:25.526899 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.527793 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.529618 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.530062 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.530189 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.530943 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:07:25.531530 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:07:25.531664 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.531798 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.531883 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:07:25.542299 10438 raft_consensus.cc:1261] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Refusing update from remote peer 666c40aa069c417b88c8add306391578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:25.542299 10574 raft_consensus.cc:1261] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Refusing update from remote peer 666c40aa069c417b88c8add306391578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:25.543678 10650 consensus_queue.cc:1035] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:25.544160 10645 consensus_queue.cc:1035] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:25.556850 10654 mvcc.cc:205] Tried to move back new op lower bound from 7315485677728919552 to 7315485677518151680. Current Snapshot: MvccSnapshot[applied={T|T < 7315485677728919552}]
I20260806 09:07:25.574386 10156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:25.574777 10156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:25.579670 10156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Sending DeleteTablet for 3 replicas of tablet c824b227d81f41049b6d377686c9c76d
I20260806 09:07:25.581120 10554 tablet_service.cc:1514] Processing DeleteTablet for tablet c824b227d81f41049b6d377686c9c76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:25 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:33610
I20260806 09:07:25.581326 10281 tablet_service.cc:1514] Processing DeleteTablet for tablet c824b227d81f41049b6d377686c9c76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:25 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:46678
I20260806 09:07:25.581369 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet c824b227d81f41049b6d377686c9c76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:25 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:46898
I20260806 09:07:25.582669 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10218
I20260806 09:07:25.593861 10663 tablet_replica.cc:330] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: stopping tablet replica
I20260806 09:07:25.594843 10663 raft_consensus.cc:2229] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:25.595811 10665 tablet_replica.cc:330] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: stopping tablet replica
I20260806 09:07:25.596606 10663 raft_consensus.cc:2258] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:25.597400 10665 raft_consensus.cc:2229] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:25.600796 10663 ts_tablet_manager.cc:1905] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:25.601630 10665 raft_consensus.cc:2258] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:25.604692 10664 tablet_replica.cc:330] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: stopping tablet replica
I20260806 09:07:25.606215 10665 ts_tablet_manager.cc:1905] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:25.606709 10664 raft_consensus.cc:2229] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 1 LEADER]: Raft consensus shutting down.
Aug 06 09:07:25 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:07:25.611721 10663 ts_tablet_manager.cc:1918] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:25.611935 10663 log.cc:1196] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/c824b227d81f41049b6d377686c9c76d
I20260806 09:07:25.607504 10664 raft_consensus.cc:2258] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:25.612659 10663 ts_tablet_manager.cc:1939] T c824b227d81f41049b6d377686c9c76d P fd8b192e40ad4544a09ec8e5b481bf43: Deleting consensus metadata
I20260806 09:07:25.614820 10141 catalog_manager.cc:4841] TS fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963): tablet c824b227d81f41049b6d377686c9c76d (table test-table [id=be3d49392ac54b4fb6bb3e6de5dc40ff]) successfully deleted
I20260806 09:07:25.615217 10664 ts_tablet_manager.cc:1905] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:25.617898 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.584028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.27.187.129:38595)
0806 09:07:25.584266 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.584273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.591966 (+ 7693us) server_negotiation.cc:407] Connection header received
0806 09:07:25.592264 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.592272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.592375 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.592517 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:25.597751 (+ 5234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.599141 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.601853 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.602616 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.602762 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.608995 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:25.609014 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:25.609032 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:25.609100 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:25.613694 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.614940 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.614951 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.614955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.615155 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.615672 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.615678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.615682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.616179 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:25.616498 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.617194 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.617499 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:07:25.618698 10665 ts_tablet_manager.cc:1918] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:25.618891 10665 log.cc:1196] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/c824b227d81f41049b6d377686c9c76d
I20260806 09:07:25.619469 10665 ts_tablet_manager.cc:1939] T c824b227d81f41049b6d377686c9c76d P ef604b59e0814ec683979c83bfcffc8f: Deleting consensus metadata
I20260806 09:07:25.619715 10342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:25.621618 10143 catalog_manager.cc:4841] TS ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163): tablet c824b227d81f41049b6d377686c9c76d (table test-table [id=be3d49392ac54b4fb6bb3e6de5dc40ff]) successfully deleted
I20260806 09:07:25.732344 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10218
I20260806 09:07:25.754601 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10353
W20260806 09:07:25.754854 10143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:38595 (error 108)
I20260806 09:07:25.757058 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.756503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46716)
0806 09:07:25.756732 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.756876 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 09:07:25.757365 10143 catalog_manager.cc:4584] TS 666c40aa069c417b88c8add306391578 (127.27.187.129:38595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c824b227d81f41049b6d377686c9c76d: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Aug 06 09:07:25 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:07:25.774369 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.756225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.27.187.130:36163)
0806 09:07:25.756426 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.756431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.756448 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:07:25.756506 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.756509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.756625 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.756751 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:25.758422 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.759710 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.761241 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.761764 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.761912 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.767473 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:25.767493 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:25.767511 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:25.767558 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:25.771054 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.772056 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.772064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.772066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.772243 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.772675 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.772678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.772680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.773097 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:25.773364 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.773774 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.773960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:07:25.775321 10479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:25.811380 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.810772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46730)
0806 09:07:25.811010 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.811130 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:07:25.812633 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.812234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46738)
0806 09:07:25.812395 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.812496 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 09:07:25.873667 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.873150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46744)
0806 09:07:25.873413 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.873503 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 09:07:25.874974 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.874520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46750)
0806 09:07:25.874743 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.874836 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:07:25.877796 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10353
I20260806 09:07:25.896869 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10490
Aug 06 09:07:25 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:07:25.915715 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.897720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.27.187.131:33963)
0806 09:07:25.897907 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:25.897913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:25.897932 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:07:25.898098 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:25.898102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:25.898178 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:25.898277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:25.899916 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.900835 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.902392 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:25.902955 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:25.903067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:25.908492 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:25.908510 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:25.908526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:25.908577 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:25.912310 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.913291 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.913301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.913303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.913469 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:25.913944 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:25.913948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:25.913951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:25.914425 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:25.914752 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:25.915161 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 09:07:25.915422 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:07:25.916469 10615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:25.956557 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.955929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46758)
0806 09:07:25.956202 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.956326 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 09:07:25.957980 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:25.957453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46772)
0806 09:07:25.957705 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:25.957815 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260806 09:07:26.015022 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10490
I20260806 09:07:26.034745 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10124
I20260806 09:07:26.036123 10186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:26.154983 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:26.153950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46780)
0806 09:07:26.154460 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:26.154615 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304}
I20260806 09:07:26.155042 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10124
I20260806 09:07:26.180946 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--webserver_interface=127.27.187.190
--webserver_port=39801
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:26.557391 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:26.557732 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:26.557897 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:26.566301 10673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:26.566454 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:26.566569 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:26.566694 10673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:26.566784 10673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:26.577584 10673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=39801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:26.579139 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:26.580708 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:26.613785 10679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:26.613768 10682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:26.613814 10680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:26.616698 10673 server_base.cc:1034] running on GCE node
I20260806 09:07:26.617832 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:26.619998 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:26.621336 10673 hybrid_clock.cc:648] HybridClock initialized: now 1786007246621299 us; error 63 us; skew 500 ppm
I20260806 09:07:26.621925 10673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:26 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007246, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:26.632347 10673 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:26.634950 10673 webserver.cc:460] Webserver started at http://127.27.187.190:39801/ using document root <none> and password file <none>
I20260806 09:07:26.635974 10673 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:26.636216 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:26.643970 10673 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:07:26.647751 10689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:26.649585 10673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:07:26.649854 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "600a390d245442308c3fbc273182ae26"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "8ccbae1947c92e65a82dba21f8c86565"
server_key_iv: "294d6330ee62673f7e9118750505a1b8"
server_key_version: "encryptionkey@0"
I20260806 09:07:26.651381 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:26.674289 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:26.679751 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:26.680109 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:26.702025 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34611 every 8 connection(s)
I20260806 09:07:26.702044 10673 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34611
I20260806 09:07:26.704020 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:26.705610 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10673
I20260806 09:07:26.706460 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:38595
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=40559
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:26.712841 10741 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:07:26.716711 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap starting.
I20260806 09:07:26.738354 10741 log.cc:824] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:26.752794 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:07:26.753439 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap complete.
I20260806 09:07:26.764004 10741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:26.765262 10741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600a390d245442308c3fbc273182ae26, State: Initialized, Role: FOLLOWER
I20260806 09:07:26.766201 10741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:26.766913 10741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:26.767123 10741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:26.767410 10741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:07:26.772161 10741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:26.772867 10741 leader_election.cc:302] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600a390d245442308c3fbc273182ae26; no voters:
I20260806 09:07:26.773684 10741 leader_election.cc:288] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:07:26.773852 10745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:07:26.775307 10745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 LEADER]: Becoming Leader. State: Replica: 600a390d245442308c3fbc273182ae26, State: Running, Role: LEADER
I20260806 09:07:26.775972 10745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:26.776448 10741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:26.779053 10746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:26.779639 10746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:26.784098 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600a390d245442308c3fbc273182ae26. Latest consensus state: current_term: 2 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:26.784547 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:26.787328 10751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:26.797395 10751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:26.798986 10751 catalog_manager.cc:1259] Loaded cluster ID: 9de39faa14824caebe97c5439514247f
I20260806 09:07:26.799108 10751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:26.802577 10751 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 09:07:26.816696 10763 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:07:26.820749 10751 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 09:07:26.831117 10751 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:26.833317 10751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Loaded TSK: 0
I20260806 09:07:26.835191 10751 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:07:27.105198 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:27.105580 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:27.105764 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:27.114038 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:27.114271 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:27.114487 10743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:27.126112 10743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:38595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=40559
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:27.127691 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:27.129519 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:27.147423 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:27.147445 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:27.147516 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:27.149776 10743 server_base.cc:1034] running on GCE node
I20260806 09:07:27.150727 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:27.152217 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:27.153455 10743 hybrid_clock.cc:648] HybridClock initialized: now 1786007247153375 us; error 87 us; skew 500 ppm
I20260806 09:07:27.153847 10743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:27 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:27.157830 10743 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:27.160014 10743 webserver.cc:460] Webserver started at http://127.27.187.129:40559/ using document root <none> and password file <none>
I20260806 09:07:27.161635 10743 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:27.161877 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:27.168797 10743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:07:27.172463 10779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:27.174327 10743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:07:27.174575 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "666c40aa069c417b88c8add306391578"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "c80518564d7070e524cddb487947079b"
server_key_iv: "b61c71c8ad0f151e2a37812c479a2479"
server_key_version: "encryptionkey@0"
I20260806 09:07:27.175858 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:27.192840 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:27.198351 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:27.198755 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:27.199990 10743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:27.202200 10787 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260806 09:07:27.210875 10787 ts_tablet_manager.cc:1852] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 09:07:27.212448 10787 ts_tablet_manager.cc:1905] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:27.223605 10787 ts_tablet_manager.cc:1918] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:27.223902 10787 log.cc:1196] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/c824b227d81f41049b6d377686c9c76d
I20260806 09:07:27.224411 10787 ts_tablet_manager.cc:1939] T c824b227d81f41049b6d377686c9c76d P 666c40aa069c417b88c8add306391578: Deleting consensus metadata
I20260806 09:07:27.224840 10743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 09:07:27.225026 10743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.023s user 0.001s sys 0.000s
I20260806 09:07:27.225234 10743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:27.225340 10743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:27.264796 10743 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:38595
I20260806 09:07:27.264882 10896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:38595 every 8 connection(s)
I20260806 09:07:27.266294 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:27.270141 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10743
I20260806 09:07:27.271124 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:36163
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=38369
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:27 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:27.295004 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:27.270320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:55347)
0806 09:07:27.271480 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:27.271533 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:27.273486 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:27.275130 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:27.275143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:27.275839 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:27.276592 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:27.276614 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.278528 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.278536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:27.279961 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:27.279980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.281366 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.281375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:27.281670 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:27.282506 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:27.282543 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:27.285390 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:27.289879 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:27.289893 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:27.289907 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:27.290433 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:27.291046 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:27.291051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:27.291055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:27.291271 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:27.292172 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:27.292182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:27.292538 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 09:07:27.292844 (+ 306us) client_negotiation.cc:240] Negotiation successful
0806 09:07:27.293510 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":429,"threads_started":1}
I20260806 09:07:27.295009 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:27.269717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:55347 (local address 127.27.187.190:34611)
0806 09:07:27.270362 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:27.270382 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:27.271709 (+ 1327us) server_negotiation.cc:407] Connection header received
0806 09:07:27.274125 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:27.274138 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:27.274513 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:27.274963 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:27.276853 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.278348 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.280139 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.280942 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.281298 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:27.285728 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:27.285737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:27.285753 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:27.285797 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:27.289657 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:27.290629 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:27.290637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:27.290643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:27.290908 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:27.291416 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:27.291424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:27.291432 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:27.291965 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:27.292221 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:27.293083 (+ 862us) server_negotiation.cc:299] Negotiation successful
0806 09:07:27.293534 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
I20260806 09:07:27.298921 10897 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:27.299286 10897 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:27.300068 10897 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:27.303159 10706 ts_manager.cc:194] Registered new tserver with Master: 666c40aa069c417b88c8add306391578 (127.27.187.129:38595)
I20260806 09:07:27.305568 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:55347
W20260806 09:07:27.643421 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:27.643731 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:27.643829 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:27.652622 10901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:27.652776 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:27.652992 10901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:27.666147 10901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=38369
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:27.667740 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:27.669456 10901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:27.684654 10907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:27.684697 10909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:27.684681 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:27.686200 10901 server_base.cc:1034] running on GCE node
I20260806 09:07:27.686998 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:27.688073 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:27.689417 10901 hybrid_clock.cc:648] HybridClock initialized: now 1786007247689307 us; error 64 us; skew 500 ppm
I20260806 09:07:27.689759 10901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:27 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:27.693415 10901 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:27.695264 10901 webserver.cc:460] Webserver started at http://127.27.187.130:38369/ using document root <none> and password file <none>
I20260806 09:07:27.696045 10901 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:27.696239 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:27.703078 10901 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:07:27.707288 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:27.709371 10901 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:07:27.709597 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ef604b59e0814ec683979c83bfcffc8f"
format_stamp: "Formatted at 2026-08-06 09:07:24 on dist-test-slave-8tdl"
server_key: "3c8792c1d6549e445303cfab3655fe05"
server_key_iv: "55fa6c1b5467e7710da1fd67cea351d5"
server_key_version: "encryptionkey@0"
I20260806 09:07:27.711088 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:27.743124 10901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:27.748481 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:27.748752 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:27.749871 10901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:27.751432 10901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:27.751526 10901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:27.751602 10901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:27.751660 10901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:27.791198 10901 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36163
I20260806 09:07:27.791370 11033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36163 every 8 connection(s)
I20260806 09:07:27.792871 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:07:27.796633 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 10901
I20260806 09:07:27.797506 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:33963
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=36045
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:07:27 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:27.820374 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:27.795910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:59925 (local address 127.27.187.190:34611)
0806 09:07:27.796531 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:27.796537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:27.797596 (+ 1059us) server_negotiation.cc:407] Connection header received
0806 09:07:27.799944 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:27.799950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:27.800059 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:27.800189 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:27.801847 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.802579 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.807947 (+ 5368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.808431 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.808609 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:27.813053 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:27.813059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:27.813062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:27.813112 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:27.816168 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:27.817479 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:27.817484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:27.817486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:27.817698 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:27.818303 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:27.818307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:27.818309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:27.818613 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:27.818833 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:27.819839 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0806 09:07:27.820078 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":964,"threads_started":1}
I20260806 09:07:27.822086 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:27.796521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:59925)
0806 09:07:27.797410 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:27.797462 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:27.799612 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:27.800468 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:27.800479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:27.801038 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:27.801647 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:27.801666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.802805 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.802810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:27.807698 (+ 4888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:27.807726 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:27.808727 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:27.808738 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:27.809034 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:27.809831 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:27.809858 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:27.812712 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:27.816410 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:27.816423 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:27.816449 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:27.817258 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:27.817844 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:27.817852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:27.817859 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:27.818149 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:27.818805 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:27.818817 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:27.819349 (+ 532us) client_negotiation.cc:769] Sending connection context
0806 09:07:27.819691 (+ 342us) client_negotiation.cc:240] Negotiation successful
0806 09:07:27.820379 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":378,"threads_started":1}
I20260806 09:07:27.824179 11034 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:27.824584 11034 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:27.825814 11034 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:27.827852 10706 ts_manager.cc:194] Registered new tserver with Master: ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163)
I20260806 09:07:27.829051 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:59925
W20260806 09:07:28.189088 11038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:28.189458 11038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:28.189636 11038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:28.198017 11038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:28.198271 11038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:28.198496 11038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:28.209446 11038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=36045
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:28.211027 11038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:28.212445 11038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:28.227916 11046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:28.228055 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:28.228017 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:28.228724 11038 server_base.cc:1034] running on GCE node
I20260806 09:07:28.229447 11038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:28.231052 11038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:28.232468 11038 hybrid_clock.cc:648] HybridClock initialized: now 1786007248232427 us; error 43 us; skew 500 ppm
I20260806 09:07:28.232872 11038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:28 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:28.236735 11038 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:28.238543 11038 webserver.cc:460] Webserver started at http://127.27.187.131:36045/ using document root <none> and password file <none>
I20260806 09:07:28.239415 11038 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:28.239544 11038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:28.245827 11038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:07:28.249312 11053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:28.251066 11038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 09:07:28.251364 11038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
format_stamp: "Formatted at 2026-08-06 09:07:25 on dist-test-slave-8tdl"
server_key: "cc885f9897008cb57f33d50cc9f558b2"
server_key_iv: "f3d92c7079220b4d010b2bf037680082"
server_key_version: "encryptionkey@0"
I20260806 09:07:28.252497 11038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:28.272746 11038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:28.278126 11038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:28.278447 11038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:28.279780 11038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:28.281344 11038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:28.281472 11038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:28.281615 11038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:28.281713 11038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:28.309508 10897 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
I20260806 09:07:28.319403 11038 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33963
I20260806 09:07:28.319468 11167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33963 every 8 connection(s)
I20260806 09:07:28.320941 11038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:28.322623 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11038
Aug 06 09:07:28 dist-test-slave-8tdl krb5kdc[10111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:28.345521 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.323750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.27.187.190:34611)
0806 09:07:28.327231 (+ 3481us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.327241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.329484 (+ 2243us) server_negotiation.cc:407] Connection header received
0806 09:07:28.329577 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.329583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.329694 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.329813 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:28.332254 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.333042 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.334593 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.335081 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.335232 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.338276 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:28.338281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:28.338284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:28.338318 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:28.342090 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.343197 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.343202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.343206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.343374 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.343855 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.343864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.343871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.344259 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:28.344470 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.344916 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.345131 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3287,"thread_start_us":3137,"threads_started":1}
I20260806 09:07:28.347946 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.327143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:35085 (local address 127.27.187.190:34611)
0806 09:07:28.328735 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.328742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.328764 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:07:28.330893 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.330898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.330978 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.331073 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:28.333092 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.333908 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.335634 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.336071 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.336242 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.340031 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:28.340036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:28.340039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:28.340122 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:28.344082 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.345190 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.345194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.345197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.345329 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.345958 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.345965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.345971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.346260 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:28.346449 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.347379 (+ 930us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.347576 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1310,"threads_started":1}
I20260806 09:07:28.349100 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.324372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:35085)
0806 09:07:28.327874 (+ 3502us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.327926 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.330490 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.331422 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.331436 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.332275 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:28.332897 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.332918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.334097 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.334101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.335480 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.335493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.336363 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.336371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.336611 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.337291 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:28.337315 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:28.339803 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:28.344243 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:28.344251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:28.344268 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:28.345033 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:28.345466 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:28.345471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:28.345475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:28.345793 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:28.346414 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:28.346420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:28.346896 (+ 476us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.347206 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.347802 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3189,"thread_start_us":3008,"threads_started":1}
I20260806 09:07:28.350659 11168 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34611
I20260806 09:07:28.351033 11168 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:28.351775 11168 heartbeater.cc:511] Master 127.27.187.190:34611 requested a full tablet report, sending...
I20260806 09:07:28.353343 10706 ts_manager.cc:194] Registered new tserver with Master: fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963)
I20260806 09:07:28.354377 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:35085
I20260806 09:07:28.361403 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:07:28.369740 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.362893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.27.187.190:34611)
0806 09:07:28.363163 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.363171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.363694 (+ 523us) server_negotiation.cc:407] Connection header received
0806 09:07:28.363981 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.363985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.364161 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.364340 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:07:28.364810 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.365571 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.367307 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.367770 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.367921 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.368479 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:07:28.369044 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:07:28.369219 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.369332 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.369427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:07:28.370962 10706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:07:28.378980 10706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:07:28.409374 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.394139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.27.187.131:33963)
0806 09:07:28.397138 (+ 2999us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.397149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.397187 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:07:28.397297 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.397305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.397587 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.399353 (+ 1766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:28.399754 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.401843 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.406276 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.407737 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.407956 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.408187 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.408838 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.409020 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2806,"thread_start_us":2633,"threads_started":1}
I20260806 09:07:28.409454 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.393628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33963 (local address 127.0.0.1:33626)
0806 09:07:28.396271 (+ 2643us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.396296 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.396453 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.398420 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.398428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.398467 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:28.399055 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.399071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.402073 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.402083 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.405994 (+ 3911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.406018 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.408374 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.408383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.408571 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.408610 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.408753 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.408994 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2478,"mutex_wait_us":894,"thread_start_us":1386,"threads_started":1}
I20260806 09:07:28.411373 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.393614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36163 (local address 127.0.0.1:46910)
0806 09:07:28.395597 (+ 1983us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.395682 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.395977 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.398046 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.398057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.398093 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:28.398716 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.398736 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.402368 (+ 3632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.402377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.407097 (+ 4720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.407114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.409880 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.409892 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.410077 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.410113 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.410212 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.410378 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"mutex_wait_us":1367,"thread_start_us":1317,"threads_started":1}
I20260806 09:07:28.411446 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.393708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.27.187.130:36163)
0806 09:07:28.397444 (+ 3736us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.397452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.397477 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:07:28.397572 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.397579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.397796 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.397993 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:28.399000 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.402057 (+ 3057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.407274 (+ 5217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.408907 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.409125 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.409310 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.410898 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.411106 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3572,"thread_start_us":3446,"threads_started":1}
I20260806 09:07:28.413694 11102 tablet_service.cc:1467] Processing CreateTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 (DEFAULT_TABLE table=test-table [id=5003397d4e874e2a90b8748762b1a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:28.415473 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cf003de6e84206a2a232b5e4ebbc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:28.415503 10966 tablet_service.cc:1467] Processing CreateTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 (DEFAULT_TABLE table=test-table [id=5003397d4e874e2a90b8748762b1a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:28.416945 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.398008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.0.0.1:46796)
0806 09:07:28.406587 (+ 8579us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.406860 (+ 273us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.407015 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.407790 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.407797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.407834 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:28.408258 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.408272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.410382 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.410387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.413619 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.413639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.415859 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.415867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.416055 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.416093 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.416215 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.416561 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8433,"thread_start_us":7655,"threads_started":1}
I20260806 09:07:28.417018 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.398616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.27.187.129:38595)
0806 09:07:28.406420 (+ 7804us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.406430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.406956 (+ 526us) server_negotiation.cc:407] Connection header received
0806 09:07:28.407219 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.407230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.407531 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.407767 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:28.408474 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.410212 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.413874 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.415448 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.415688 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.415885 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.416395 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.416589 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7592,"thread_start_us":7394,"threads_started":1}
I20260806 09:07:28.417189 10966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cf003de6e84206a2a232b5e4ebbc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:28.420457 10830 tablet_service.cc:1467] Processing CreateTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 (DEFAULT_TABLE table=test-table [id=5003397d4e874e2a90b8748762b1a6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:28.422029 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cf003de6e84206a2a232b5e4ebbc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:28.431393 11185 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Bootstrap starting.
I20260806 09:07:28.433873 11186 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Bootstrap starting.
I20260806 09:07:28.437273 11185 tablet_bootstrap.cc:654] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:28.437740 11186 tablet_bootstrap.cc:654] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:28.438815 11185 log.cc:824] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:28.439035 11186 log.cc:824] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:28.440697 11187 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Bootstrap starting.
I20260806 09:07:28.442909 11185 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: No bootstrap required, opened a new log
I20260806 09:07:28.443228 11185 ts_tablet_manager.cc:1397] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260806 09:07:28.444840 11187 tablet_bootstrap.cc:654] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:28.445322 11186 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: No bootstrap required, opened a new log
I20260806 09:07:28.445634 11186 ts_tablet_manager.cc:1397] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260806 09:07:28.446233 11187 log.cc:824] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:28.450201 11187 tablet_bootstrap.cc:492] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: No bootstrap required, opened a new log
I20260806 09:07:28.450521 11187 ts_tablet_manager.cc:1397] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 09:07:28.453872 11186 raft_consensus.cc:348] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.454321 11186 raft_consensus.cc:374] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:28.454424 11186 raft_consensus.cc:729] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b192e40ad4544a09ec8e5b481bf43, State: Initialized, Role: FOLLOWER
I20260806 09:07:28.454273 11185 raft_consensus.cc:348] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.454825 11185 raft_consensus.cc:374] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:28.454918 11185 raft_consensus.cc:729] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef604b59e0814ec683979c83bfcffc8f, State: Initialized, Role: FOLLOWER
I20260806 09:07:28.455184 11186 consensus_queue.cc:260] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.455749 11185 consensus_queue.cc:260] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.456993 11168 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
I20260806 09:07:28.457563 11034 heartbeater.cc:503] Master 127.27.187.190:34611 was elected leader, sending a full tablet report...
I20260806 09:07:28.458415 11185 ts_tablet_manager.cc:1428] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260806 09:07:28.460326 11186 ts_tablet_manager.cc:1428] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20260806 09:07:28.462909 11187 raft_consensus.cc:348] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.463419 11187 raft_consensus.cc:374] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:28.463498 11187 raft_consensus.cc:729] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666c40aa069c417b88c8add306391578, State: Initialized, Role: FOLLOWER
I20260806 09:07:28.464227 11187 consensus_queue.cc:260] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.466140 11187 ts_tablet_manager.cc:1428] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260806 09:07:28.519075 10898 tablet.cc:2369] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:07:28.545295 11035 tablet.cc:2369] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:07:28.572849 11169 tablet.cc:2369] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:07:28.646968 11192 raft_consensus.cc:482] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:07:28.647284 11192 raft_consensus.cc:504] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.649016 11192 leader_election.cc:288] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963), 666c40aa069c417b88c8add306391578 (127.27.187.129:38595)
I20260806 09:07:28.657142 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.649675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:38155 (local address 127.27.187.131:33963)
0806 09:07:28.649898 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.649904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.651154 (+ 1250us) server_negotiation.cc:407] Connection header received
0806 09:07:28.651308 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.651312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.651413 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.651511 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:28.652219 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.653017 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.654885 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.656054 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.656247 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.656345 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.656688 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.656840 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:07:28.657205 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.649657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:59443 (local address 127.27.187.129:38595)
0806 09:07:28.649898 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.649905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.650996 (+ 1091us) server_negotiation.cc:407] Connection header received
0806 09:07:28.651170 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.651175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.651272 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.651364 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:28.652072 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.652854 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.654919 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.656070 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.656262 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.656345 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.656791 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.656938 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:07:28.657485 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.649691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:38595 (local address 127.27.187.130:59443)
0806 09:07:28.650873 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.650890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.651023 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.651404 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.651409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.651435 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:28.651887 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.651898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.653017 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.653022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.654664 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.654736 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.656411 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.656422 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.656615 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.656650 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.656759 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.656903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":27,"thread_start_us":503,"threads_started":1}
I20260806 09:07:28.657509 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.649717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:33963 (local address 127.27.187.130:38155)
0806 09:07:28.651002 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:28.651023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:28.651175 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:28.651555 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:28.651559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:28.651592 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:28.652005 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.652017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.653153 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.653156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.654717 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:28.654731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.656308 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.656315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:28.656492 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.656538 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:07:28.656657 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:07:28.656900 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":32,"thread_start_us":842,"threads_started":1}
I20260806 09:07:28.658787 11122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cf003de6e84206a2a232b5e4ebbc39" candidate_uuid: "ef604b59e0814ec683979c83bfcffc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" is_pre_election: true
I20260806 09:07:28.658782 10850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cf003de6e84206a2a232b5e4ebbc39" candidate_uuid: "ef604b59e0814ec683979c83bfcffc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666c40aa069c417b88c8add306391578" is_pre_election: true
I20260806 09:07:28.659396 10850 raft_consensus.cc:2454] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef604b59e0814ec683979c83bfcffc8f in term 0.
I20260806 09:07:28.659435 11122 raft_consensus.cc:2454] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef604b59e0814ec683979c83bfcffc8f in term 0.
I20260806 09:07:28.660204 10920 leader_election.cc:302] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666c40aa069c417b88c8add306391578, ef604b59e0814ec683979c83bfcffc8f; no voters:
I20260806 09:07:28.660676 11192 raft_consensus.cc:2781] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:07:28.660840 11192 raft_consensus.cc:482] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:07:28.660981 11192 raft_consensus.cc:3037] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:28.663517 11192 raft_consensus.cc:504] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.664285 11192 leader_election.cc:288] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [CANDIDATE]: Term 1 election: Requested vote from peers fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963), 666c40aa069c417b88c8add306391578 (127.27.187.129:38595)
I20260806 09:07:28.664920 11122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cf003de6e84206a2a232b5e4ebbc39" candidate_uuid: "ef604b59e0814ec683979c83bfcffc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
I20260806 09:07:28.665201 11122 raft_consensus.cc:3037] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:28.665103 10850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cf003de6e84206a2a232b5e4ebbc39" candidate_uuid: "ef604b59e0814ec683979c83bfcffc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666c40aa069c417b88c8add306391578"
I20260806 09:07:28.665585 10850 raft_consensus.cc:3037] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:28.668179 10850 raft_consensus.cc:2454] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef604b59e0814ec683979c83bfcffc8f in term 1.
I20260806 09:07:28.668203 11122 raft_consensus.cc:2454] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef604b59e0814ec683979c83bfcffc8f in term 1.
I20260806 09:07:28.668763 10920 leader_election.cc:302] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 666c40aa069c417b88c8add306391578, ef604b59e0814ec683979c83bfcffc8f; no voters:
I20260806 09:07:28.669180 11192 raft_consensus.cc:2781] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:28.669742 11192 raft_consensus.cc:686] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 1 LEADER]: Becoming Leader. State: Replica: ef604b59e0814ec683979c83bfcffc8f, State: Running, Role: LEADER
I20260806 09:07:28.670316 11192 consensus_queue.cc:237] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } }
I20260806 09:07:28.676230 10705 catalog_manager.cc:5495] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f reported cstate change: term changed from 0 to 1, leader changed from <none> to ef604b59e0814ec683979c83bfcffc8f (127.27.187.130). New cstate: current_term: 1 leader_uuid: "ef604b59e0814ec683979c83bfcffc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef604b59e0814ec683979c83bfcffc8f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 } health_report { overall_health: UNKNOWN } } }
I20260806 09:07:28.708501 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.701828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.27.187.130:36163)
0806 09:07:28.702005 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.702009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.702141 (+ 132us) server_negotiation.cc:407] Connection header received
0806 09:07:28.702381 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.702385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.702474 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.702643 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:07:28.703181 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.703972 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.706074 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.706497 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.706605 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.707261 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:07:28.707833 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:07:28.707950 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.708101 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.708181 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 09:07:28.718066 10850 raft_consensus.cc:1261] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Refusing update from remote peer ef604b59e0814ec683979c83bfcffc8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:28.718078 11122 raft_consensus.cc:1261] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Refusing update from remote peer ef604b59e0814ec683979c83bfcffc8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:28.719113 11192 consensus_queue.cc:1035] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8b192e40ad4544a09ec8e5b481bf43" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:28.719426 11196 consensus_queue.cc:1035] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "666c40aa069c417b88c8add306391578" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:28.731846 11199 mvcc.cc:205] Tried to move back new op lower bound from 7315485690736627712 to 7315485690559705088. Current Snapshot: MvccSnapshot[applied={T|T < 7315485690736627712}]
I20260806 09:07:28.737809 11201 mvcc.cc:205] Tried to move back new op lower bound from 7315485690736627712 to 7315485690559705088. Current Snapshot: MvccSnapshot[applied={T|T < 7315485690736627712}]
I20260806 09:07:28.750334 10705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:28.750662 10705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:28.754774 10705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Sending DeleteTablet for 3 replicas of tablet 48cf003de6e84206a2a232b5e4ebbc39
I20260806 09:07:28.756500 11102 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:28 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:33626
I20260806 09:07:28.756500 10830 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:28 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:46796
I20260806 09:07:28.756837 10966 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cf003de6e84206a2a232b5e4ebbc39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:28 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:46910
I20260806 09:07:28.757797 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10743
I20260806 09:07:28.766603 11210 tablet_replica.cc:330] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: stopping tablet replica
I20260806 09:07:28.767580 11210 raft_consensus.cc:2229] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:28.768327 11210 raft_consensus.cc:2258] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:28.768932 11209 tablet_replica.cc:330] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: stopping tablet replica
I20260806 09:07:28.769644 11211 tablet_replica.cc:330] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: stopping tablet replica
I20260806 09:07:28.770207 11209 raft_consensus.cc:2229] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:28.770821 11211 raft_consensus.cc:2229] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:07:28.771469 11209 raft_consensus.cc:2258] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:28.771577 11210 ts_tablet_manager.cc:1905] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:28.772194 11211 raft_consensus.cc:2258] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:28.774660 11211 ts_tablet_manager.cc:1905] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:28.775368 11209 ts_tablet_manager.cc:1905] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:28.779770 11210 ts_tablet_manager.cc:1918] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:28.779948 11210 log.cc:1196] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/48cf003de6e84206a2a232b5e4ebbc39
I20260806 09:07:28.780584 11210 ts_tablet_manager.cc:1939] T 48cf003de6e84206a2a232b5e4ebbc39 P 666c40aa069c417b88c8add306391578: Deleting consensus metadata
I20260806 09:07:28.782706 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.758814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.27.187.129:38595)
0806 09:07:28.759059 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.759068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.759207 (+ 139us) server_negotiation.cc:407] Connection header received
0806 09:07:28.759442 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.759451 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.759562 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.759716 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:28.762022 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.763736 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.766518 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.767578 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.767767 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.771906 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:28.771920 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:28.771937 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:28.771981 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:28.776279 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.777704 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.777716 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.777721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.777970 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.780621 (+ 2651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.780627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.780631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.781229 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:28.781768 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.781951 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.782257 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:07:28.782868 10693 catalog_manager.cc:4841] TS 666c40aa069c417b88c8add306391578 (127.27.187.129:38595): tablet 48cf003de6e84206a2a232b5e4ebbc39 (table test-table [id=5003397d4e874e2a90b8748762b1a6b4]) successfully deleted
I20260806 09:07:28.783313 11211 ts_tablet_manager.cc:1918] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:28.783387 10891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:28.783528 11211 log.cc:1196] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/48cf003de6e84206a2a232b5e4ebbc39
I20260806 09:07:28.784178 11211 ts_tablet_manager.cc:1939] T 48cf003de6e84206a2a232b5e4ebbc39 P ef604b59e0814ec683979c83bfcffc8f: Deleting consensus metadata
I20260806 09:07:28.786139 10693 catalog_manager.cc:4841] TS ef604b59e0814ec683979c83bfcffc8f (127.27.187.130:36163): tablet 48cf003de6e84206a2a232b5e4ebbc39 (table test-table [id=5003397d4e874e2a90b8748762b1a6b4]) successfully deleted
I20260806 09:07:28.787101 11209 ts_tablet_manager.cc:1918] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:28.787405 11209 log.cc:1196] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/48cf003de6e84206a2a232b5e4ebbc39
I20260806 09:07:28.788049 11209 ts_tablet_manager.cc:1939] T 48cf003de6e84206a2a232b5e4ebbc39 P fd8b192e40ad4544a09ec8e5b481bf43: Deleting consensus metadata
I20260806 09:07:28.790949 10691 catalog_manager.cc:4841] TS fd8b192e40ad4544a09ec8e5b481bf43 (127.27.187.131:33963): tablet 48cf003de6e84206a2a232b5e4ebbc39 (table test-table [id=5003397d4e874e2a90b8748762b1a6b4]) successfully deleted
I20260806 09:07:28.899537 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10743
I20260806 09:07:28.928328 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10901
I20260806 09:07:28.947979 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:28.929293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.27.187.130:36163)
0806 09:07:28.929516 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:28.929521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:28.929624 (+ 103us) server_negotiation.cc:407] Connection header received
0806 09:07:28.929848 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:28.929853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:28.929940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:28.930050 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:28.932183 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.933048 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.935488 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:28.935996 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:28.936129 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:28.940285 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:28.940305 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:28.940321 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:28.940379 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:28.944077 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.945271 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.945280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.945285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.945477 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:28.945986 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:28.945990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:28.945992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:28.946484 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:28.946884 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:28.947372 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 09:07:28.947572 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:07:28.948802 11028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:29.072896 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10901
I20260806 09:07:29.096926 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11038
I20260806 09:07:29.113003 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:29.097994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.27.187.131:33963)
0806 09:07:29.098238 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:29.098243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:29.098288 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:07:29.098497 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:29.098501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:29.098568 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:29.098664 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:29.100452 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:29.101227 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:29.102757 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:29.103168 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:29.103276 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:29.106275 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:29.106291 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:29.106308 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:29.106353 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:29.109631 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:29.110598 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:29.110606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:29.110609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:29.110902 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:29.111384 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:29.111388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:29.111390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:29.111809 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:29.112032 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:29.112536 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 09:07:29.112732 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:07:29.113839 11163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:29.209694 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11038
I20260806 09:07:29.229339 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 10673
I20260806 09:07:29.230886 10736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:29.356618 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 10673
I20260806 09:07:29.383083 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--webserver_interface=127.27.187.190
--webserver_port=39801
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:29.758389 11216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:29.758903 11216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:29.759074 11216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:29.768204 11216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:29.768369 11216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:29.768478 11216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:29.768579 11216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:29.768709 11216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:29.780488 11216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=39801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:29.781999 11216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:29.783497 11216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:29.797482 11226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:29.797535 11223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:29.797680 11224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:29.798738 11216 server_base.cc:1034] running on GCE node
I20260806 09:07:29.799685 11216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:29.801438 11216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:29.802801 11216 hybrid_clock.cc:648] HybridClock initialized: now 1786007249802727 us; error 83 us; skew 500 ppm
I20260806 09:07:29.803220 11216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:29 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007249, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:29.807130 11216 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:29.809079 11216 webserver.cc:460] Webserver started at http://127.27.187.190:39801/ using document root <none> and password file <none>
I20260806 09:07:29.809907 11216 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:29.810077 11216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:29.816552 11216 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 09:07:29.822634 11233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:29.824635 11216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:07:29.824901 11216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "600a390d245442308c3fbc273182ae26"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "8ccbae1947c92e65a82dba21f8c86565"
server_key_iv: "294d6330ee62673f7e9118750505a1b8"
server_key_version: "encryptionkey@0"
I20260806 09:07:29.826149 11216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:29.861562 11216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:29.867513 11216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:29.867848 11216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:29.890553 11216 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34611
I20260806 09:07:29.890576 11286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34611 every 8 connection(s)
I20260806 09:07:29.892410 11216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:29.896370 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11216
I20260806 09:07:29.897284 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:38595
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=40559
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:29.900961 11287 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:07:29.908185 11287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap starting.
I20260806 09:07:29.941654 11287 log.cc:824] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:29.965648 11287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:07:29.966480 11287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26: Bootstrap complete.
I20260806 09:07:29.982816 11287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:29.984270 11287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600a390d245442308c3fbc273182ae26, State: Initialized, Role: FOLLOWER
I20260806 09:07:29.985119 11287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:29.985437 11287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:29.985577 11287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:29.985775 11287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:07:29.992216 11287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:29.992952 11287 leader_election.cc:302] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600a390d245442308c3fbc273182ae26; no voters:
I20260806 09:07:29.993722 11287 leader_election.cc:288] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:07:29.993927 11291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:07:29.995440 11291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [term 3 LEADER]: Becoming Leader. State: Replica: 600a390d245442308c3fbc273182ae26, State: Running, Role: LEADER
I20260806 09:07:29.996112 11287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:29.996369 11291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } }
I20260806 09:07:29.999581 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600a390d245442308c3fbc273182ae26. Latest consensus state: current_term: 3 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:30.000068 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:30.001101 11292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "600a390d245442308c3fbc273182ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600a390d245442308c3fbc273182ae26" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34611 } } }
I20260806 09:07:30.001534 11292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600a390d245442308c3fbc273182ae26 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:30.011929 11298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:30.025223 11298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:30.027158 11298 catalog_manager.cc:1259] Loaded cluster ID: 9de39faa14824caebe97c5439514247f
I20260806 09:07:30.027348 11298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 09:07:30.029934 11298 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb1b89c19ad google::LogMessage::Fail() at ??:0
@ 0x7fb1b89c488a google::LogMessage::SendToLog() at ??:0
@ 0x7fb1b89c14de google::LogMessage::Flush() at ??:0
@ 0x7fb1b89c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb1ce3f4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb1ce3f1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb1ce479610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb1ce479439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1bb60d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1bb61091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb1bb610799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1bb5ebeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb1be8c56db start_thread at ??:0
@ 0x7fb1b785b61f clone at ??:0
W20260806 09:07:30.402457 11289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:30.402878 11289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:30.403090 11289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:30.411340 11289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:30.411474 11289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:30.411676 11289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:30.422330 11289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:38595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=40559
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:30.423874 11289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:30.425390 11289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:30.440461 11328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:30.440629 11329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:30.440563 11331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:30.441466 11289 server_base.cc:1034] running on GCE node
I20260806 09:07:30.442214 11289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:30.443490 11289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:30.444757 11289 hybrid_clock.cc:648] HybridClock initialized: now 1786007250444702 us; error 51 us; skew 500 ppm
I20260806 09:07:30.445151 11289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:30 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:30.448800 11289 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:30.450488 11289 webserver.cc:460] Webserver started at http://127.27.187.129:40559/ using document root <none> and password file <none>
I20260806 09:07:30.451297 11289 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:30.451419 11289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:30.457597 11289 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:07:30.461269 11338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:30.463454 11289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:07:30.463693 11289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "666c40aa069c417b88c8add306391578"
format_stamp: "Formatted at 2026-08-06 09:07:23 on dist-test-slave-8tdl"
server_key: "c80518564d7070e524cddb487947079b"
server_key_iv: "b61c71c8ad0f151e2a37812c479a2479"
server_key_version: "encryptionkey@0"
I20260806 09:07:30.464824 11289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:30.486996 11289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:30.492825 11289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:30.493189 11289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:30.494489 11289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:30.496218 11289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:30.496320 11289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:30.496450 11289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:30.496536 11289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:30.536864 11289 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:38595
I20260806 09:07:30.537005 11454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:38595 every 8 connection(s)
I20260806 09:07:30.538506 11289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:30.539925 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11289
I20260806 09:07:30.540807 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:36163
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=38369
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:30.545044 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:30.542643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:59649)
0806 09:07:30.543700 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:30.543996 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":438,"threads_started":1}
W20260806 09:07:30.545976 11455 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:34611 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:34611: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
I20260806 09:07:30.547302 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:30.546862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:36827)
0806 09:07:30.547048 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:30.547141 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:07:30.548483 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:30.547986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:40521)
0806 09:07:30.548182 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:30.548306 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:07:30.548811 11455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:07:30.917661 11458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:30.917991 11458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:30.918144 11458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:30.925907 11458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:30.926048 11458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:30.926255 11458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:30.936810 11458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=38369
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:30.938313 11458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:30.939829 11458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:30.955101 11463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:30.955214 11464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:30.955217 11466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:30.956123 11458 server_base.cc:1034] running on GCE node
I20260806 09:07:30.956879 11458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:30.958003 11458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:30.959301 11458 hybrid_clock.cc:648] HybridClock initialized: now 1786007250959232 us; error 61 us; skew 500 ppm
I20260806 09:07:30.959724 11458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:30 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:30.963735 11458 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:30.965689 11458 webserver.cc:460] Webserver started at http://127.27.187.130:38369/ using document root <none> and password file <none>
I20260806 09:07:30.966511 11458 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:30.966711 11458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:30.973403 11458 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 09:07:30.976766 11473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:30.978489 11458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:07:30.978744 11458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "ef604b59e0814ec683979c83bfcffc8f"
format_stamp: "Formatted at 2026-08-06 09:07:24 on dist-test-slave-8tdl"
server_key: "3c8792c1d6549e445303cfab3655fe05"
server_key_iv: "55fa6c1b5467e7710da1fd67cea351d5"
server_key_version: "encryptionkey@0"
I20260806 09:07:30.979861 11458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:30.996542 11458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:31.001866 11458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:31.002228 11458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:31.003509 11458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:31.005849 11458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:31.005973 11458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:31.006111 11458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:31.006556 11458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:31.044895 11458 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36163
I20260806 09:07:31.044993 11587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36163 every 8 connection(s)
I20260806 09:07:31.046526 11458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:07:31.052585 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.050382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:59951)
0806 09:07:31.051317 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.051570 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":418,"threads_started":1}
W20260806 09:07:31.053362 11588 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:34611 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:34611: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
I20260806 09:07:31.054286 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.053941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:45669)
0806 09:07:31.054094 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.054168 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 09:07:31.055356 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.054915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:45597)
0806 09:07:31.055094 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.055231 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 09:07:31.055677 11588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:07:31.056169 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11458
I20260806 09:07:31.056767 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:33963
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:34611
--webserver_port=36045
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:31.413566 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:31.413908 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:31.414083 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:31.421962 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:31.422093 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:31.422283 11591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:31.432993 11591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=36045
--tserver_master_addrs=127.27.187.190:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:31.434473 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:31.435923 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:31.450942 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:31.450971 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:31.451054 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:31.451764 11591 server_base.cc:1034] running on GCE node
I20260806 09:07:31.452533 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:31.453648 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:31.454898 11591 hybrid_clock.cc:648] HybridClock initialized: now 1786007251454853 us; error 32 us; skew 500 ppm
I20260806 09:07:31.455294 11591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:31 dist-test-slave-8tdl krb5kdc[10111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007251, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:31.458834 11591 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:31.460594 11591 webserver.cc:460] Webserver started at http://127.27.187.131:36045/ using document root <none> and password file <none>
I20260806 09:07:31.461396 11591 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:31.461563 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:31.468163 11591 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:07:31.471932 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:31.473825 11591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:07:31.474076 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "fd8b192e40ad4544a09ec8e5b481bf43"
format_stamp: "Formatted at 2026-08-06 09:07:25 on dist-test-slave-8tdl"
server_key: "cc885f9897008cb57f33d50cc9f558b2"
server_key_iv: "f3d92c7079220b4d010b2bf037680082"
server_key_version: "encryptionkey@0"
I20260806 09:07:31.475378 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:31.514978 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:31.520238 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:31.520556 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:31.521643 11591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:31.523263 11591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:31.523397 11591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:31.523538 11591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:31.523629 11591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:31.550536 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.549528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:59435)
0806 09:07:31.550115 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.550272 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
I20260806 09:07:31.563370 11591 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:33963
I20260806 09:07:31.563469 11721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:33963 every 8 connection(s)
I20260806 09:07:31.564997 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:31.569700 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11591
I20260806 09:07:31.571127 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.568539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:58729)
0806 09:07:31.569499 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.569815 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":326,"threads_started":1}
W20260806 09:07:31.572100 11722 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:34611 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:34611: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
I20260806 09:07:31.573088 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.572742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:48927)
0806 09:07:31.572894 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.572968 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 09:07:31.574059 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:31.573745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:56147)
0806 09:07:31.573874 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:31.573951 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:07:31.574470 11722 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:07:32.057761 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:32.056534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:60263)
0806 09:07:32.057243 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:32.057440 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":310,"threads_started":1}
I20260806 09:07:32.552759 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:32.551733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:42363)
0806 09:07:32.552480 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:32.552580 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":473,"threads_started":1}
I20260806 09:07:32.576156 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:32.575271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:43019)
0806 09:07:32.575784 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:32.575914 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
I20260806 09:07:33.059800 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:33.058980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:56421)
0806 09:07:33.059514 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:33.059617 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":275,"threads_started":1}
I20260806 09:07:33.555112 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:33.553989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:53305)
0806 09:07:33.554740 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:33.554854 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":451,"threads_started":1}
I20260806 09:07:33.578184 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:33.577369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:60871)
0806 09:07:33.577917 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:33.578000 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
I20260806 09:07:34.061717 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:34.060916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:50797)
0806 09:07:34.061438 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:34.061532 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
I20260806 09:07:34.557202 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:34.556298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:46859)
0806 09:07:34.556895 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:34.556991 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":276,"threads_started":1}
I20260806 09:07:34.580399 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:34.579504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:56939)
0806 09:07:34.580074 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:34.580202 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
I20260806 09:07:35.063832 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:35.062959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:52485)
0806 09:07:35.063526 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:35.063631 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":320,"threads_started":1}
I20260806 09:07:35.559523 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:35.558399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:54819)
0806 09:07:35.559138 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:35.559278 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":448,"threads_started":1}
I20260806 09:07:35.582362 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:35.581483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:50923)
0806 09:07:35.582080 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:35.582179 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":306,"threads_started":1}
I20260806 09:07:36.066116 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:36.065269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:36677)
0806 09:07:36.065822 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:36.065921 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":272,"threads_started":1}
I20260806 09:07:36.561555 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:36.560667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:50943)
0806 09:07:36.561230 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:36.561369 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260806 09:07:36.584419 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:36.583540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:59323)
0806 09:07:36.584034 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:36.584130 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260806 09:07:37.068120 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:37.067270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:35517)
0806 09:07:37.067845 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:37.067941 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":338,"threads_started":1}
I20260806 09:07:37.563910 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:37.562817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:33333)
0806 09:07:37.563524 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:37.563630 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":383,"threads_started":1}
I20260806 09:07:37.586764 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:37.585634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:39743)
0806 09:07:37.586377 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:37.586478 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":312,"threads_started":1}
I20260806 09:07:38.070495 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:38.069540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:40037)
0806 09:07:38.070118 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:38.070220 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
I20260806 09:07:38.566092 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:38.565178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:56269)
0806 09:07:38.565754 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:38.565903 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":306,"threads_started":1}
I20260806 09:07:38.589222 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:38.588052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:45937)
0806 09:07:38.588783 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:38.588891 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":271,"threads_started":1}
I20260806 09:07:39.072665 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:39.071784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:57635)
0806 09:07:39.072361 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:39.072471 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260806 09:07:39.568208 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:39.567309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:38865)
0806 09:07:39.567902 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:39.568005 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":262,"threads_started":1}
I20260806 09:07:39.591521 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:39.590635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:53629)
0806 09:07:39.591245 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:39.591345 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":260,"threads_started":1}
I20260806 09:07:40.074926 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:40.073911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:38793)
0806 09:07:40.074521 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:40.074644 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":298,"threads_started":1}
I20260806 09:07:40.570453 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:40.569452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:59825)
0806 09:07:40.570041 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:40.570270 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
I20260806 09:07:40.593518 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:40.592669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:41819)
0806 09:07:40.593215 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:40.593338 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
I20260806 09:07:41.076870 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:41.076046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:51529)
0806 09:07:41.076591 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:41.076690 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260806 09:07:41.572537 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:41.571673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:32785)
0806 09:07:41.572231 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:41.572345 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260806 09:07:41.595469 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:41.594550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:51331)
0806 09:07:41.595187 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:41.595297 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":411,"threads_started":1}
I20260806 09:07:42.078958 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:42.077949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:51947)
0806 09:07:42.078581 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:42.078762 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":388,"threads_started":1}
I20260806 09:07:42.574393 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:42.573661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:42123)
0806 09:07:42.574123 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:42.574220 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260806 09:07:42.597307 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:42.596520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:50725)
0806 09:07:42.596999 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:42.597085 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260806 09:07:43.081017 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:43.080070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:53777)
0806 09:07:43.080687 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:43.080830 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":356,"threads_started":1}
I20260806 09:07:43.576488 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:43.575631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:49229)
0806 09:07:43.576168 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:43.576275 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":266,"threads_started":1}
I20260806 09:07:43.599367 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:43.598518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:50847)
0806 09:07:43.599081 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:43.599172 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":343,"threads_started":1}
I20260806 09:07:44.083345 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:44.082301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:37977)
0806 09:07:44.082956 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:44.083126 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":296,"threads_started":1}
I20260806 09:07:44.578436 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:44.577611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:46959)
0806 09:07:44.578152 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:44.578244 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":292,"threads_started":1}
I20260806 09:07:44.601562 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:44.600612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:55925)
0806 09:07:44.601226 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:44.601355 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":338,"threads_started":1}
I20260806 09:07:45.085453 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:45.084452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:48675)
0806 09:07:45.085090 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:45.085262 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20260806 09:07:45.580806 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:45.579860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:33809)
0806 09:07:45.580443 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:45.580553 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":297,"threads_started":1}
I20260806 09:07:45.603452 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:45.602608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:42365)
0806 09:07:45.603143 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:45.603235 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":306,"threads_started":1}
I20260806 09:07:46.087951 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.087034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.130:53429)
0806 09:07:46.087617 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:46.087736 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":339,"threads_started":1}
I20260806 09:07:46.580390 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11289
I20260806 09:07:46.583364 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.581856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.129:58185)
0806 09:07:46.582794 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:46.583040 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710,"spinlock_wait_cycles":27392,"thread_start_us":694,"threads_started":1}
I20260806 09:07:46.603061 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.581706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.27.187.129:38595)
0806 09:07:46.583006 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:46.583027 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:46.583077 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:07:46.585265 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:46.585276 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:46.585814 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:46.586174 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:46.588084 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.589644 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.591126 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.591881 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.592059 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:46.595370 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:46.595380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:46.595399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:46.595460 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:46.598982 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.599891 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.599898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.599906 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.600100 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.600695 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.600705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.600707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.601334 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:46.601701 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:46.602181 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 09:07:46.602746 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":683,"threads_started":1}
I20260806 09:07:46.604664 11450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:46.606562 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.604763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34611 (local address 127.27.187.131:54225)
0806 09:07:46.606091 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:46.606258 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":920,"threads_started":1}
I20260806 09:07:46.694825 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11289
I20260806 09:07:46.715456 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11458
I20260806 09:07:46.735955 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.716462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.27.187.130:36163)
0806 09:07:46.717050 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:46.717067 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:46.717108 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:07:46.718920 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:46.718932 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:46.719395 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:46.719780 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:46.721551 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.723088 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.724511 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.725253 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.725490 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:46.728515 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:46.728523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:46.728540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:46.728584 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:46.731913 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.732858 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.732872 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.732880 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.733084 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.733545 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.733551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.733554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.734190 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:46.734539 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:46.735176 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 09:07:46.735637 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
I20260806 09:07:46.737602 11583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:46.826606 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11458
I20260806 09:07:46.849648 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11591
I20260806 09:07:46.871982 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:46.850987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.27.187.131:33963)
0806 09:07:46.851747 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:46.851763 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:46.851811 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:07:46.853766 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:46.853779 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:46.854277 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:46.854751 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:46.856567 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.858167 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.859782 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:46.860562 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:46.860805 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:46.864239 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:46.864250 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:46.864270 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:46.864321 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:46.868039 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.868938 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.868947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.868954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.869148 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:46.869584 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:46.869588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:46.869590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:46.870212 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:46.870610 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:46.871141 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 09:07:46.871659 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":395,"threads_started":1}
I20260806 09:07:46.873675 11717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:46.969935 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11591
2026-08-06T09:07:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25671 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:07:47 dist-test-slave-8tdl krb5kdc[11779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:07:47 dist-test-slave-8tdl krb5kdc[11779](info): set up 2 sockets
Aug 06 09:07:47 dist-test-slave-8tdl krb5kdc[11779](info): commencing operation
krb5kdc: starting...
W20260806 09:07:48.139142 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:07:48 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:07:48.475932 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:48.857944 11792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:48.858368 11792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:48.858531 11792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:48.867581 11792 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:07:48.867789 11792 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:07:48.867915 11792 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:07:48.868021 11792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:07:48.868156 11792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:48.868283 11792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:48.868388 11792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:07:48.868481 11792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:07:48.880764 11792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:48.882499 11792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:48.884027 11792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:48.898594 11798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:48.898622 11797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:48.898540 11800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:48.899659 11792 server_base.cc:1034] running on GCE node
I20260806 09:07:48.900527 11792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:48.902206 11792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:48.903498 11792 hybrid_clock.cc:648] HybridClock initialized: now 1786007268903397 us; error 66 us; skew 500 ppm
I20260806 09:07:48.903908 11792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:48 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:48.908051 11792 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:07:48.910076 11792 webserver.cc:460] Webserver started at http://127.27.187.190:36875/ using document root <none> and password file <none>
I20260806 09:07:48.911017 11792 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:48.911150 11792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:48.911556 11792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:48.915259 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "77180c5858cd4bb99fd6a351c4f2faa8"
format_stamp: "Formatted at 2026-08-06 09:07:48 on dist-test-slave-8tdl"
server_key: "3117e21f48b7fb05b444c37f549d6392"
server_key_iv: "93cf6ac2869c85be77e64e19c5b7d094"
server_key_version: "encryptionkey@0"
I20260806 09:07:48.916183 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "77180c5858cd4bb99fd6a351c4f2faa8"
format_stamp: "Formatted at 2026-08-06 09:07:48 on dist-test-slave-8tdl"
server_key: "3117e21f48b7fb05b444c37f549d6392"
server_key_iv: "93cf6ac2869c85be77e64e19c5b7d094"
server_key_version: "encryptionkey@0"
I20260806 09:07:48.922477 11792 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:07:48.927146 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:48.929284 11792 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:07:48.929569 11792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "77180c5858cd4bb99fd6a351c4f2faa8"
format_stamp: "Formatted at 2026-08-06 09:07:48 on dist-test-slave-8tdl"
server_key: "3117e21f48b7fb05b444c37f549d6392"
server_key_iv: "93cf6ac2869c85be77e64e19c5b7d094"
server_key_version: "encryptionkey@0"
I20260806 09:07:48.929904 11792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:48.960304 11792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:48.965508 11792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:48.965848 11792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:48.987888 11792 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:40813
I20260806 09:07:48.987910 11859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:40813 every 8 connection(s)
I20260806 09:07:48.989480 11792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:07:48.991279 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11792
I20260806 09:07:48.991523 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:07:48.992448 28398 external_mini_cluster.cc:1428] Setting key 1b3dc835629dd12f9e6ee9557eb749b8
I20260806 09:07:48.995036 11860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:49.012346 11860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap starting.
Aug 06 09:07:49 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:49.016501 11860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:49.017781 11860 log.cc:824] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:49.021106 11860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: No bootstrap required, opened a new log
I20260806 09:07:49.022598 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:48.994974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.27.187.190:40813)
0806 09:07:48.995927 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:48.995940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:48.995982 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:07:48.997738 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:48.997766 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:48.998543 (+ 777us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:48.999037 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:49.001209 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.002975 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.005383 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.006897 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.007134 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:49.013975 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:49.013986 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:49.014003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:49.014059 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:49.017810 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:49.018752 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:49.018761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:49.018769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:49.018994 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:49.019460 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:49.019466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:49.019469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:49.020080 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:49.020431 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:49.020925 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 09:07:49.021477 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":453,"threads_started":1}
I20260806 09:07:49.030108 11860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:07:49.030747 11860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:49.030893 11860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Initialized, Role: FOLLOWER
I20260806 09:07:49.031652 11860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:07:49.031919 11860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:07:49.032104 11860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:07:49.032300 11860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:49.034919 11860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:07:49.035624 11860 leader_election.cc:302] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77180c5858cd4bb99fd6a351c4f2faa8; no voters:
I20260806 09:07:49.036461 11860 leader_election.cc:288] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:07:49.036652 11865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:49.037659 11865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 LEADER]: Becoming Leader. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Running, Role: LEADER
I20260806 09:07:49.038420 11865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:07:49.039472 11860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:07:49.041692 11866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:07:49.042232 11866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:49.041811 11867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77180c5858cd4bb99fd6a351c4f2faa8. Latest consensus state: current_term: 1 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:07:49.043006 11867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:07:49.047256 11874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:07:49.054201 11874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:07:49.066618 11874 catalog_manager.cc:1347] Generated new cluster ID: 28318c1e83bc4314842c794feb290c70
I20260806 09:07:49.066862 11874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:07:49.088044 11874 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:07:49.089917 11874 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:07:49.102360 11874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Generated new TSK 0
I20260806 09:07:49.103621 11874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:07:49.191689 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:49.583004 11886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:49.583410 11886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:49.583606 11886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:49.591475 11886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:49.591670 11886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:49.591872 11886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:07:49.602901 11886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:49.604477 11886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:49.605957 11886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:49.626595 11894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:49.626557 11892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:49.626528 11891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:49.627959 11886 server_base.cc:1034] running on GCE node
I20260806 09:07:49.628713 11886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:49.629850 11886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:49.631105 11886 hybrid_clock.cc:648] HybridClock initialized: now 1786007269631050 us; error 37 us; skew 500 ppm
I20260806 09:07:49.631541 11886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:49 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:49.635573 11886 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:07:49.637324 11886 webserver.cc:460] Webserver started at http://127.27.187.129:37659/ using document root <none> and password file <none>
I20260806 09:07:49.638082 11886 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:49.638207 11886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:49.638499 11886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:49.642076 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "70018f3b149240398798fdcddd320b81"
format_stamp: "Formatted at 2026-08-06 09:07:49 on dist-test-slave-8tdl"
server_key: "2b4aaa579fdc1d14bbb2cf4846451abe"
server_key_iv: "3f454cf7d9d1400b9e1ac3c26e45ba79"
server_key_version: "encryptionkey@0"
I20260806 09:07:49.642984 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "70018f3b149240398798fdcddd320b81"
format_stamp: "Formatted at 2026-08-06 09:07:49 on dist-test-slave-8tdl"
server_key: "2b4aaa579fdc1d14bbb2cf4846451abe"
server_key_iv: "3f454cf7d9d1400b9e1ac3c26e45ba79"
server_key_version: "encryptionkey@0"
I20260806 09:07:49.648959 11886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:07:49.653126 11901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:49.654896 11886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:07:49.655064 11886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "70018f3b149240398798fdcddd320b81"
format_stamp: "Formatted at 2026-08-06 09:07:49 on dist-test-slave-8tdl"
server_key: "2b4aaa579fdc1d14bbb2cf4846451abe"
server_key_iv: "3f454cf7d9d1400b9e1ac3c26e45ba79"
server_key_version: "encryptionkey@0"
I20260806 09:07:49.655287 11886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:49.679508 11886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:49.684701 11886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:49.684993 11886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:49.686028 11886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:49.687569 11886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:49.687659 11886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:49.687758 11886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:49.687803 11886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:49.729768 11886 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43255
I20260806 09:07:49.729851 12013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43255 every 8 connection(s)
I20260806 09:07:49.731184 11886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:07:49.739473 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 11886
I20260806 09:07:49.739738 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:07:49.740593 28398 external_mini_cluster.cc:1428] Setting key 0160807db5f6373e9198e5626c6f3094
Aug 06 09:07:49 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:49.758867 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:49.734505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:43425 (local address 127.27.187.190:40813)
0806 09:07:49.735049 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:49.735054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:49.736209 (+ 1155us) server_negotiation.cc:407] Connection header received
0806 09:07:49.738300 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:49.738308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:49.738453 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:49.738582 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:49.741105 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.742049 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.744122 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.744901 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.745080 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:49.750073 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:49.750084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:49.750092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:49.750149 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:49.754268 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:49.755676 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:49.755685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:49.755691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:49.755864 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:49.756712 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:49.756717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:49.756719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:49.757025 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:49.757149 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:49.758272 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0806 09:07:49.758477 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":246,"threads_started":1}
I20260806 09:07:49.760562 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:49.735216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:43425)
0806 09:07:49.736034 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:49.736085 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:49.737966 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:49.738903 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:49.738918 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:49.739824 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:49.740830 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:49.740869 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.742235 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.742240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:49.743937 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:49.743953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:49.745426 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:49.745433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:49.745731 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:49.746758 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:49.746791 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:49.749669 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:49.754501 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:49.754510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:49.754528 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:49.755494 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:49.756040 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:49.756044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:49.756047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:49.756512 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:49.757202 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:49.757215 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:49.757782 (+ 567us) client_negotiation.cc:769] Sending connection context
0806 09:07:49.758135 (+ 353us) client_negotiation.cc:240] Negotiation successful
0806 09:07:49.758844 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":359,"threads_started":1}
I20260806 09:07:49.762596 12014 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:07:49.762998 12014 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:49.763983 12014 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:07:49.766865 11824 ts_manager.cc:194] Registered new tserver with Master: 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255)
I20260806 09:07:49.768447 11824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:43425
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:07:49.818357 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:07:50.184362 12020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:50.184689 12020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:50.184867 12020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:50.193109 12020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:50.193266 12020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:50.193491 12020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:07:50.204610 12020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:50.206101 12020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:50.207578 12020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:50.222479 12025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:50.222441 12027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:50.222515 12029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:50.223569 12020 server_base.cc:1034] running on GCE node
I20260806 09:07:50.224306 12020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:50.225349 12020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:50.226624 12020 hybrid_clock.cc:648] HybridClock initialized: now 1786007270226568 us; error 31 us; skew 500 ppm
I20260806 09:07:50.227043 12020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:50 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:50.230623 12020 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:07:50.232252 12020 webserver.cc:460] Webserver started at http://127.27.187.130:35927/ using document root <none> and password file <none>
I20260806 09:07:50.232988 12020 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:50.233098 12020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:50.233394 12020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:50.236640 12020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "bc75f05864744604a399396a8f14e356"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "52efdd3f053fcd73281502eab7fbd66a"
server_key_iv: "2cbf8a718918ba234a5fc71702a037d2"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.237423 12020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "bc75f05864744604a399396a8f14e356"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "52efdd3f053fcd73281502eab7fbd66a"
server_key_iv: "2cbf8a718918ba234a5fc71702a037d2"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.243394 12020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:07:50.248006 12036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.249806 12020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:07:50.250067 12020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "bc75f05864744604a399396a8f14e356"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "52efdd3f053fcd73281502eab7fbd66a"
server_key_iv: "2cbf8a718918ba234a5fc71702a037d2"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.250329 12020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:50.282269 12020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:50.287719 12020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:50.288059 12020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:50.289213 12020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:50.290781 12020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:50.290870 12020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.291024 12020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:50.291077 12020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.328925 12020 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38385
I20260806 09:07:50.328984 12149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38385 every 8 connection(s)
I20260806 09:07:50.330343 12020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:07:50.332445 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12020
I20260806 09:07:50.332677 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:07:50.333294 28398 external_mini_cluster.cc:1428] Setting key 78c5f7152f15e759023f28c09dd1fc40
Aug 06 09:07:50 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:50.357539 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:50.333520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:53291 (local address 127.27.187.190:40813)
0806 09:07:50.334039 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:50.334044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:50.335376 (+ 1332us) server_negotiation.cc:407] Connection header received
0806 09:07:50.337570 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:50.337579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:50.337680 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:50.337814 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:50.339735 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.340955 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.342979 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.343798 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.343956 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:50.348616 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:50.348628 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:50.348631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:50.348672 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:50.353115 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:50.354436 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:50.354442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:50.354444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:50.354638 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:50.355415 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:50.355420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:50.355422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:50.355799 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:50.355945 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:50.357029 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0806 09:07:50.357256 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":300,"threads_started":1}
I20260806 09:07:50.359236 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:50.334177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:53291)
0806 09:07:50.335196 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:50.335248 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:50.337248 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:50.338037 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:50.338046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:50.338635 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:50.339529 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:50.339554 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.341185 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.341195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:50.342788 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:50.342803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.344074 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.344079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:50.344338 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:50.345234 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:50.345261 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:50.348240 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:50.353349 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:50.353362 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:50.353386 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:50.354213 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:50.354847 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:50.354855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:50.354861 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:50.355160 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:50.355959 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:50.355971 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:50.356529 (+ 558us) client_negotiation.cc:769] Sending connection context
0806 09:07:50.356879 (+ 350us) client_negotiation.cc:240] Negotiation successful
0806 09:07:50.357545 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":430,"threads_started":1}
I20260806 09:07:50.361049 12150 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:07:50.361406 12150 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:50.362195 12150 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:07:50.363824 11824 ts_manager.cc:194] Registered new tserver with Master: bc75f05864744604a399396a8f14e356 (127.27.187.130:38385)
I20260806 09:07:50.364670 11824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:53291
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:07:50.411906 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:07:50.773111 12014 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
W20260806 09:07:50.806202 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:07:50.806532 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:07:50.806634 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:07:50.814930 12156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:07:50.815074 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:07:50.815198 12156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:07:50.825578 12156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:07:50.827003 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:07:50.828406 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:07:50.844798 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:50.844791 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:07:50.845070 12165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:07:50.845703 12156 server_base.cc:1034] running on GCE node
I20260806 09:07:50.846446 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:07:50.847786 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:07:50.849145 12156 hybrid_clock.cc:648] HybridClock initialized: now 1786007270849006 us; error 63 us; skew 500 ppm
I20260806 09:07:50.849571 12156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:07:50 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:07:50.853865 12156 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:07:50.856122 12156 webserver.cc:460] Webserver started at http://127.27.187.131:36537/ using document root <none> and password file <none>
I20260806 09:07:50.856950 12156 fs_manager.cc:359] Metadata directory not provided
I20260806 09:07:50.857079 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:07:50.857440 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:07:50.861248 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "9c9e4397cb213b68bfd6087b101512b8"
server_key_iv: "2e417393127c8171eb70d05e2625dfe5"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.862195 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "9c9e4397cb213b68bfd6087b101512b8"
server_key_iv: "2e417393127c8171eb70d05e2625dfe5"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.869869 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260806 09:07:50.874490 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.876426 12156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:07:50.876665 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "9c9e4397cb213b68bfd6087b101512b8"
server_key_iv: "2e417393127c8171eb70d05e2625dfe5"
server_key_version: "encryptionkey@0"
I20260806 09:07:50.876919 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:07:50.896811 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:07:50.901889 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:07:50.902235 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:07:50.903398 12156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:07:50.904978 12156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:07:50.905062 12156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.905229 12156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:07:50.905282 12156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:07:50.944798 12156 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45025
I20260806 09:07:50.944860 12286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45025 every 8 connection(s)
I20260806 09:07:50.946363 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:07:50.948133 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12156
I20260806 09:07:50.948354 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:07:50.948993 28398 external_mini_cluster.cc:1428] Setting key b6b469bde10b114295fc22513a3f3892
Aug 06 09:07:50 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:07:50.971927 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:50.950209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:52675 (local address 127.27.187.190:40813)
0806 09:07:50.950945 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:50.950951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:50.951848 (+ 897us) server_negotiation.cc:407] Connection header received
0806 09:07:50.953972 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:50.953976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:50.954051 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:50.954151 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:50.956335 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.957776 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.959937 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.960494 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.960617 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:50.964724 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:50.964729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:50.964733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:50.964773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:50.968328 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:50.969255 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:50.969267 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:50.969270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:50.969402 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:50.969894 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:50.969898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:50.969900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:50.970204 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:50.970383 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:50.971252 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 09:07:50.971505 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":845,"threads_started":1}
I20260806 09:07:50.972962 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:50.950707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:52675)
0806 09:07:50.951652 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:50.951719 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:50.953628 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:50.954542 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:50.954553 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:50.955360 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:07:50.956096 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:50.956120 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.958037 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.958043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:50.959734 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:50.959752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:50.960752 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:50.960759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:50.961001 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:50.961917 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:07:50.961950 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:07:50.964446 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:07:50.968530 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:50.968541 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:50.968563 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:50.969103 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:50.969527 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:07:50.969532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:07:50.969536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:07:50.969755 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:07:50.970348 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:07:50.970357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:07:50.970839 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 09:07:50.971144 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 09:07:50.971755 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":395,"threads_started":1}
I20260806 09:07:50.974584 12287 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:07:50.974980 12287 heartbeater.cc:464] Registering TS with master...
I20260806 09:07:50.976155 12287 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:07:50.978519 11824 ts_manager.cc:194] Registered new tserver with Master: 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
I20260806 09:07:50.979560 11824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:52675
I20260806 09:07:50.991564 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:07:51.018436 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:50.998367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.27.187.190:40813)
0806 09:07:50.998643 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:50.998794 (+ 151us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:50.999283 (+ 489us) server_negotiation.cc:407] Connection header received
0806 09:07:50.999505 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:50.999510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:50.999644 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:50.999801 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:51.001680 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.002486 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.004377 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.004810 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.004932 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.009548 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:51.009554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:51.009558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:51.009607 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:51.014002 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.015561 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.015567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.015569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.015767 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.016452 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.016463 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.016466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.016901 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:51.017153 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.017781 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.018054 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:07:51.024358 11824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:07:51.031865 11824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:07:51.059372 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.048351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.27.187.130:38385)
0806 09:07:51.050181 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.050193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.050231 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:07:51.050355 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.050364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.050706 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.050960 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:51.051780 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.053464 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.056861 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.057995 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.058219 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.058445 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.058862 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.059040 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1541,"threads_started":1}
I20260806 09:07:51.059581 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.048153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38385 (local address 127.0.0.1:45558)
0806 09:07:51.049167 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:51.049229 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:51.049483 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:51.050975 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:51.050984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:51.051018 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:51.051537 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.051562 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.053667 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.053673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.056670 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.056685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.058266 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.058274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.058498 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.058560 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 09:07:51.058755 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 09:07:51.058997 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":1,"thread_start_us":387,"threads_started":1}
I20260806 09:07:51.063223 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.050331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:40872)
0806 09:07:51.052841 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:51.052864 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:51.053012 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:51.054009 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:51.054017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:51.054040 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:51.054572 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.054593 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.056652 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.056658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.059510 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.059534 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.062191 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.062198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.062409 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.062461 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 09:07:51.062545 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 09:07:51.062753 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2377,"mutex_wait_us":2,"thread_start_us":2203,"threads_started":1}
I20260806 09:07:51.063803 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.049222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45025 (local address 127.0.0.1:48830)
0806 09:07:51.052419 (+ 3197us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:51.052441 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:51.052590 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:51.053642 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:51.053650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:51.053678 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:51.054250 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.054264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.057037 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.057045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.060119 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.060136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.062655 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.062663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.062871 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.062909 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:07:51.063016 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:07:51.063161 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3046,"thread_start_us":2818,"threads_started":1}
I20260806 09:07:51.063822 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.049951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.27.187.131:45025)
0806 09:07:51.051831 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.051842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.052752 (+ 910us) server_negotiation.cc:407] Connection header received
0806 09:07:51.052877 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.052888 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.053148 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.053345 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:51.054500 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.056244 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.060398 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.062378 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.062606 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.062811 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.063097 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.063423 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"thread_start_us":1501,"threads_started":1}
I20260806 09:07:51.063923 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.049928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.27.187.129:43255)
0806 09:07:51.050635 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.050645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.053170 (+ 2525us) server_negotiation.cc:407] Connection header received
0806 09:07:51.053308 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.053316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.053543 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.053763 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:51.054885 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.056492 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.059994 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.061606 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.061819 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.062010 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.063394 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.063577 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":518,"threads_started":1}
I20260806 09:07:51.064137 12084 tablet_service.cc:1467] Processing CreateTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (DEFAULT_TABLE table=test-table [id=c149c9d843d1413890a7ada795ea95f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:51.066038 12084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fedeabbf62d41d1ae23bc9ca81e3538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:51.068418 11949 tablet_service.cc:1467] Processing CreateTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (DEFAULT_TABLE table=test-table [id=c149c9d843d1413890a7ada795ea95f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:51.070168 11949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fedeabbf62d41d1ae23bc9ca81e3538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:51.073627 12220 tablet_service.cc:1467] Processing CreateTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (DEFAULT_TABLE table=test-table [id=c149c9d843d1413890a7ada795ea95f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:07:51.075389 12220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fedeabbf62d41d1ae23bc9ca81e3538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:07:51.079030 12306 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Bootstrap starting.
I20260806 09:07:51.082917 12306 tablet_bootstrap.cc:654] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:51.084072 12306 log.cc:824] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:51.088778 12308 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Bootstrap starting.
I20260806 09:07:51.090591 12306 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: No bootstrap required, opened a new log
I20260806 09:07:51.091017 12306 ts_tablet_manager.cc:1397] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260806 09:07:51.092592 12309 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Bootstrap starting.
I20260806 09:07:51.094054 12308 tablet_bootstrap.cc:654] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:51.095762 12308 log.cc:824] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:51.096851 12309 tablet_bootstrap.cc:654] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Neither blocks nor log segments found. Creating new log.
I20260806 09:07:51.098580 12309 log.cc:824] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:07:51.100328 12308 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: No bootstrap required, opened a new log
I20260806 09:07:51.100674 12308 ts_tablet_manager.cc:1397] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 09:07:51.101840 12309 tablet_bootstrap.cc:492] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: No bootstrap required, opened a new log
I20260806 09:07:51.102164 12309 ts_tablet_manager.cc:1397] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 09:07:51.103152 12306 raft_consensus.cc:348] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.103677 12306 raft_consensus.cc:374] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:51.103780 12306 raft_consensus.cc:729] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc75f05864744604a399396a8f14e356, State: Initialized, Role: FOLLOWER
I20260806 09:07:51.104687 12306 consensus_queue.cc:260] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.107861 12150 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
I20260806 09:07:51.109488 12306 ts_tablet_manager.cc:1428] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260806 09:07:51.113564 12309 raft_consensus.cc:348] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.114125 12308 raft_consensus.cc:348] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.114846 12309 raft_consensus.cc:374] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:51.114859 12308 raft_consensus.cc:374] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:07:51.115002 12309 raft_consensus.cc:729] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fc57e5ef6041dcb11dd6fe5aedc2cc, State: Initialized, Role: FOLLOWER
I20260806 09:07:51.115006 12308 raft_consensus.cc:729] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70018f3b149240398798fdcddd320b81, State: Initialized, Role: FOLLOWER
I20260806 09:07:51.115775 12308 consensus_queue.cc:260] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.115757 12309 consensus_queue.cc:260] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.117440 12287 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
I20260806 09:07:51.118101 12309 ts_tablet_manager.cc:1428] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260806 09:07:51.120218 12308 ts_tablet_manager.cc:1428] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
I20260806 09:07:51.123173 12315 raft_consensus.cc:482] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:07:51.123628 12315 raft_consensus.cc:504] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.126472 12315 leader_election.cc:288] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
I20260806 09:07:51.135394 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.126817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45025 (local address 127.27.187.129:35067)
0806 09:07:51.127495 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:51.127519 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:51.127694 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:51.128157 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:51.128168 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:51.128198 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:51.128936 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.128958 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.130208 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.130216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.132816 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.132835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.134707 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.134716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.134875 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.134903 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:07:51.134988 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 09:07:51.135095 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":351,"threads_started":1}
I20260806 09:07:51.135443 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.127243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:35067 (local address 127.27.187.131:45025)
0806 09:07:51.127466 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.127471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.127693 (+ 222us) server_negotiation.cc:407] Connection header received
0806 09:07:51.127839 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.127845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.127930 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.128056 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:51.129172 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.130034 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.133033 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.134369 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.134611 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.134746 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.135041 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.135194 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:07:51.136883 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fedeabbf62d41d1ae23bc9ca81e3538" candidate_uuid: "70018f3b149240398798fdcddd320b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" is_pre_election: true
I20260806 09:07:51.137583 12241 raft_consensus.cc:2454] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70018f3b149240398798fdcddd320b81 in term 0.
I20260806 09:07:51.138370 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.126847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38385 (local address 127.27.187.129:37829)
0806 09:07:51.131289 (+ 4442us) negotiation.cc:107] Waiting for socket to connect
0806 09:07:51.131307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:07:51.131437 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:07:51.131795 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:07:51.131799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:07:51.131822 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:07:51.132218 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.132229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.133505 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.133511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.135652 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:07:51.135669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.137549 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.137555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:07:51.137712 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.137735 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:07:51.137847 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:07:51.137968 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4307,"thread_start_us":4076,"threads_started":1}
I20260806 09:07:51.138367 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:37829 (local address 127.27.187.130:38385)
0806 09:07:51.127556 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.127564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.131432 (+ 3868us) server_negotiation.cc:407] Connection header received
0806 09:07:51.131559 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.131565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.131657 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.131763 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:07:51.132399 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.133340 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.135865 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.137063 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.137206 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.137308 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.137924 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.138103 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:07:51.138455 11905 leader_election.cc:302] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70018f3b149240398798fdcddd320b81, 83fc57e5ef6041dcb11dd6fe5aedc2cc; no voters:
I20260806 09:07:51.139031 12315 raft_consensus.cc:2781] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:07:51.139221 12315 raft_consensus.cc:482] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:07:51.139350 12315 raft_consensus.cc:3037] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:51.139866 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fedeabbf62d41d1ae23bc9ca81e3538" candidate_uuid: "70018f3b149240398798fdcddd320b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc75f05864744604a399396a8f14e356" is_pre_election: true
I20260806 09:07:51.140563 12104 raft_consensus.cc:2454] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70018f3b149240398798fdcddd320b81 in term 0.
I20260806 09:07:51.141935 12315 raft_consensus.cc:504] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.142958 12315 leader_election.cc:288] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 1 election: Requested vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
I20260806 09:07:51.143229 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fedeabbf62d41d1ae23bc9ca81e3538" candidate_uuid: "70018f3b149240398798fdcddd320b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc75f05864744604a399396a8f14e356"
I20260806 09:07:51.143451 12104 raft_consensus.cc:3037] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:51.143615 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fedeabbf62d41d1ae23bc9ca81e3538" candidate_uuid: "70018f3b149240398798fdcddd320b81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
I20260806 09:07:51.143868 12241 raft_consensus.cc:3037] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:07:51.147334 12104 raft_consensus.cc:2454] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70018f3b149240398798fdcddd320b81 in term 1.
I20260806 09:07:51.147393 12241 raft_consensus.cc:2454] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70018f3b149240398798fdcddd320b81 in term 1.
I20260806 09:07:51.147974 11904 leader_election.cc:302] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70018f3b149240398798fdcddd320b81, bc75f05864744604a399396a8f14e356; no voters:
I20260806 09:07:51.148396 12315 raft_consensus.cc:2781] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:07:51.149049 12315 raft_consensus.cc:686] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 1 LEADER]: Becoming Leader. State: Replica: 70018f3b149240398798fdcddd320b81, State: Running, Role: LEADER
I20260806 09:07:51.149588 12315 consensus_queue.cc:237] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:07:51.155081 11824 catalog_manager.cc:5495] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70018f3b149240398798fdcddd320b81 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "70018f3b149240398798fdcddd320b81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } health_report { overall_health: UNKNOWN } } }
I20260806 09:07:51.196101 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.190032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.27.187.129:43255)
0806 09:07:51.190263 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.190270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.190378 (+ 108us) server_negotiation.cc:407] Connection header received
0806 09:07:51.190720 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.190724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.190829 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.191001 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:07:51.191423 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.192203 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.193704 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.194140 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.194288 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.194892 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:07:51.195442 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:07:51.195630 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.195716 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.195810 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 09:07:51.198563 12288 tablet.cc:2369] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:07:51.206070 12241 raft_consensus.cc:1261] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Refusing update from remote peer 70018f3b149240398798fdcddd320b81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:51.206066 12104 raft_consensus.cc:1261] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Refusing update from remote peer 70018f3b149240398798fdcddd320b81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:07:51.207453 12315 consensus_queue.cc:1035] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:51.207837 12318 consensus_queue.cc:1035] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:07:51.221166 12321 mvcc.cc:205] Tried to move back new op lower bound from 7315485782846439424 to 7315485782633758720. Current Snapshot: MvccSnapshot[applied={T|T < 7315485782846439424}]
I20260806 09:07:51.235571 11824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:51.235865 11824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:07:51.240160 11824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Sending DeleteTablet for 3 replicas of tablet 5fedeabbf62d41d1ae23bc9ca81e3538
I20260806 09:07:51.241659 12220 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:51 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:48830
I20260806 09:07:51.241794 12084 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:51 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:45558
I20260806 09:07:51.242967 11949 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fedeabbf62d41d1ae23bc9ca81e3538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:07:51 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:40872
I20260806 09:07:51.243146 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11886
I20260806 09:07:51.254279 12332 tablet_replica.cc:330] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: stopping tablet replica
I20260806 09:07:51.256295 12332 raft_consensus.cc:2229] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:07:51.257206 12332 raft_consensus.cc:2258] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:51.261313 12332 ts_tablet_manager.cc:1905] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:51.271221 12332 ts_tablet_manager.cc:1918] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:51.271483 12332 log.cc:1196] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/5fedeabbf62d41d1ae23bc9ca81e3538
I20260806 09:07:51.271971 12331 tablet_replica.cc:330] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: stopping tablet replica
I20260806 09:07:51.272240 12332 ts_tablet_manager.cc:1939] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 70018f3b149240398798fdcddd320b81: Deleting consensus metadata
I20260806 09:07:51.272526 12331 raft_consensus.cc:2229] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:51.273123 12331 raft_consensus.cc:2258] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:51.274372 11810 catalog_manager.cc:4841] TS 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255): tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (table test-table [id=c149c9d843d1413890a7ada795ea95f3]) successfully deleted
I20260806 09:07:51.278771 12330 tablet_replica.cc:330] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: stopping tablet replica
I20260806 09:07:51.279268 12331 ts_tablet_manager.cc:1905] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:07:51.279271 12330 raft_consensus.cc:2229] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:07:51.279829 12330 raft_consensus.cc:2258] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:07:51.282266 12330 ts_tablet_manager.cc:1905] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:07:51 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:07:51.287338 12331 ts_tablet_manager.cc:1918] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:51.287541 12331 log.cc:1196] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/5fedeabbf62d41d1ae23bc9ca81e3538
I20260806 09:07:51.288087 12331 ts_tablet_manager.cc:1939] T 5fedeabbf62d41d1ae23bc9ca81e3538 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting consensus metadata
I20260806 09:07:51.289945 12330 ts_tablet_manager.cc:1918] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:07:51.290130 12330 log.cc:1196] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/5fedeabbf62d41d1ae23bc9ca81e3538
I20260806 09:07:51.290249 11811 catalog_manager.cc:4841] TS 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (table test-table [id=c149c9d843d1413890a7ada795ea95f3]) successfully deleted
I20260806 09:07:51.290797 12330 ts_tablet_manager.cc:1939] T 5fedeabbf62d41d1ae23bc9ca81e3538 P bc75f05864744604a399396a8f14e356: Deleting consensus metadata
I20260806 09:07:51.292904 11810 catalog_manager.cc:4841] TS bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): tablet 5fedeabbf62d41d1ae23bc9ca81e3538 (table test-table [id=c149c9d843d1413890a7ada795ea95f3]) successfully deleted
I20260806 09:07:51.294158 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.244806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.27.187.129:43255)
0806 09:07:51.245054 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.245064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.268270 (+ 23206us) server_negotiation.cc:407] Connection header received
0806 09:07:51.268868 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.268876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.268975 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.269052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:51.273582 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.274818 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.276401 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.277118 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.277264 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.284853 (+ 7589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:51.284879 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:51.284902 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:51.284970 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:51.289884 (+ 4914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.291387 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.291399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.291402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.291623 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.292201 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.292206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.292209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.292746 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:51.293197 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.293469 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.293759 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:07:51.295174 12009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:51.401906 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11886
I20260806 09:07:51.425104 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12020
Aug 06 09:07:51 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:07:51.444183 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.426049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.27.187.130:38385)
0806 09:07:51.426231 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.426235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.426253 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:07:51.426310 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.426314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.426390 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.426498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:51.428251 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.429165 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.430835 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.431404 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.431533 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.436809 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:51.436830 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:51.436848 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:51.436898 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:51.440851 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.441827 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.441836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.441839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.442016 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.442509 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.442513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.442516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.443104 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:51.443437 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.443704 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.443902 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:07:51.445149 12145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:51.544139 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12020
I20260806 09:07:51.564780 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12156
Aug 06 09:07:51 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:07:51.585829 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:07:51.565889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.27.187.131:45025)
0806 09:07:51.566140 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:07:51.566146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:07:51.566172 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:07:51.566364 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:07:51.566370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:07:51.566441 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:07:51.566563 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:07:51.568090 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.568932 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.571520 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:07:51.571975 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:07:51.572145 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:07:51.577552 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:07:51.577574 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:07:51.577591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:07:51.577652 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:07:51.581564 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.583160 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.583170 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.583173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.583367 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:07:51.583925 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:07:51.583930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:07:51.583933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:07:51.584481 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:07:51.584762 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 09:07:51.585275 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 09:07:51.585498 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:07:51.586773 12282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:51.685197 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12156
I20260806 09:07:51.705564 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 11792
I20260806 09:07:51.706965 11855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:07:51.834988 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 11792
I20260806 09:08:11.862200 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--webserver_interface=127.27.187.190
--webserver_port=36875
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:08:12.246564 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:12.247032 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:12.247216 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:12.255015 12338 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:08:12.255177 12338 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:08:12.255280 12338 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:08:12.255378 12338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:08:12.255482 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:12.255607 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:12.255698 12338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:08:12.255780 12338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:08:12.266552 12338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=36875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:12.268019 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:12.269474 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:12.297973 12345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:12.297909 12347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:12.298133 12344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:12.303731 12338 server_base.cc:1034] running on GCE node
I20260806 09:08:12.304939 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:12.306973 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:12.308315 12338 hybrid_clock.cc:648] HybridClock initialized: now 1786007292308213 us; error 56 us; skew 500 ppm
I20260806 09:08:12.308812 12338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:12 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:12.317930 12338 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:08:12.320098 12338 webserver.cc:460] Webserver started at http://127.27.187.190:36875/ using document root <none> and password file <none>
I20260806 09:08:12.321067 12338 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:12.321228 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:12.328331 12338 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:12.331985 12354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:12.333817 12338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:12.334076 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "77180c5858cd4bb99fd6a351c4f2faa8"
format_stamp: "Formatted at 2026-08-06 09:07:48 on dist-test-slave-8tdl"
server_key: "3117e21f48b7fb05b444c37f549d6392"
server_key_iv: "93cf6ac2869c85be77e64e19c5b7d094"
server_key_version: "encryptionkey@0"
I20260806 09:08:12.335378 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:12.349203 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:12.354529 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:12.354957 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:12.377368 12338 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:40813
I20260806 09:08:12.377399 12405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:40813 every 8 connection(s)
I20260806 09:08:12.379287 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:08:12.387511 12406 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:08:12.388649 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12338
I20260806 09:08:12.389658 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:43255
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=37659
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:08:12.391619 12406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap starting.
I20260806 09:08:12.414546 12406 log.cc:824] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:12.428264 12406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:08:12.428877 12406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap complete.
I20260806 09:08:12.440575 12406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:12.441584 12406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Initialized, Role: FOLLOWER
I20260806 09:08:12.442251 12406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:12.442524 12406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:08:12.442663 12406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:08:12.442901 12406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:08:12.447288 12406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:12.447898 12406 leader_election.cc:302] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77180c5858cd4bb99fd6a351c4f2faa8; no voters:
I20260806 09:08:12.448633 12406 leader_election.cc:288] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:08:12.448792 12410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:08:12.451031 12410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 LEADER]: Becoming Leader. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Running, Role: LEADER
I20260806 09:08:12.451431 12406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:08:12.451783 12410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:12.455245 12412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77180c5858cd4bb99fd6a351c4f2faa8. Latest consensus state: current_term: 2 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:08:12.455890 12412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:12.455245 12411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:08:12.456786 12411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:12.468228 12418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:08:12.478478 12418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:08:12.481088 12418 catalog_manager.cc:1259] Loaded cluster ID: 28318c1e83bc4314842c794feb290c70
I20260806 09:08:12.481375 12418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:08:12.487457 12418 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:08:12.491253 12418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Loaded TSK: 0
I20260806 09:08:12.533442 12427 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:08:12.547641 12418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Generated new TSK 1
I20260806 09:08:12.548065 12418 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:08:12.808926 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:12.809484 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:12.809686 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:12.818631 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:12.818881 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:12.819077 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:08:12.830008 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:43255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=37659
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:12.831610 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:12.833107 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:12.848453 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:12.849107 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:12.849366 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:12.849748 12408 server_base.cc:1034] running on GCE node
I20260806 09:08:12.850504 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:12.851755 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:12.852986 12408 hybrid_clock.cc:648] HybridClock initialized: now 1786007292852942 us; error 46 us; skew 500 ppm
I20260806 09:08:12.853349 12408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:12 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:12.857071 12408 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:08:12.859311 12408 webserver.cc:460] Webserver started at http://127.27.187.129:37659/ using document root <none> and password file <none>
I20260806 09:08:12.860121 12408 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:12.860307 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:12.866582 12408 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:08:12.869831 12443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:12.871498 12408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:08:12.871686 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "70018f3b149240398798fdcddd320b81"
format_stamp: "Formatted at 2026-08-06 09:07:49 on dist-test-slave-8tdl"
server_key: "2b4aaa579fdc1d14bbb2cf4846451abe"
server_key_iv: "3f454cf7d9d1400b9e1ac3c26e45ba79"
server_key_version: "encryptionkey@0"
I20260806 09:08:12.872787 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:12.908814 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:12.913625 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:12.913967 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:12.915318 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:12.916854 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:12.916947 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:12.917104 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:12.917191 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:12.955222 12408 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43255
I20260806 09:08:12.955273 12557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43255 every 8 connection(s)
I20260806 09:08:12.956856 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:08:12.965792 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12408
I20260806 09:08:12.966562 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:38385
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=35927
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:08:12 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:12.984501 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:12.960558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:47049)
0806 09:08:12.961396 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:12.961436 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:12.963407 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:12.964982 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:12.964991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:12.965690 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:12.966311 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:12.966332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:12.968380 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:12.968384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:12.969647 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:12.969660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:12.970770 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:12.970779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:12.971060 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:12.971803 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:12.971831 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:12.974968 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:12.979448 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:12.979457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:12.979471 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:12.980111 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:12.980792 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:12.980795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:12.980798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:12.981015 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:12.982163 (+ 1148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:12.982171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:12.982537 (+ 366us) client_negotiation.cc:769] Sending connection context
0806 09:08:12.982897 (+ 360us) client_negotiation.cc:240] Negotiation successful
0806 09:08:12.983358 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":324,"threads_started":1}
I20260806 09:08:12.985049 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:12.960225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:47049 (local address 127.27.187.190:40813)
0806 09:08:12.960801 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:12.960813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:12.961600 (+ 787us) server_negotiation.cc:407] Connection header received
0806 09:08:12.964013 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:12.964026 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:12.964441 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:12.964827 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:12.966534 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:12.968154 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:12.969807 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:12.970476 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:12.970724 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:12.975357 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:12.975367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:12.975383 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:12.975428 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:12.979216 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:12.980300 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:12.980314 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:12.980325 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:12.980583 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:12.981150 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:12.981158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:12.981164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:12.981980 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:12.982214 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:12.983178 (+ 964us) server_negotiation.cc:299] Negotiation successful
0806 09:08:12.983726 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260806 09:08:12.988301 12558 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:08:12.988657 12558 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:12.989480 12558 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:08:12.992029 12371 ts_manager.cc:194] Registered new tserver with Master: 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255)
I20260806 09:08:12.993742 12371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:47049
W20260806 09:08:13.338100 12562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:13.338490 12562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:13.338713 12562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:13.346429 12562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:13.346625 12562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:13.346841 12562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:08:13.357445 12562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:38385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=35927
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:13.358950 12562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:13.360553 12562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:13.374547 12567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:13.374583 12568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:13.374622 12570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:13.375758 12562 server_base.cc:1034] running on GCE node
I20260806 09:08:13.376391 12562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:13.377635 12562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:13.378886 12562 hybrid_clock.cc:648] HybridClock initialized: now 1786007293378800 us; error 49 us; skew 500 ppm
I20260806 09:08:13.379495 12562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:13 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:13.383060 12562 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:08:13.384788 12562 webserver.cc:460] Webserver started at http://127.27.187.130:35927/ using document root <none> and password file <none>
I20260806 09:08:13.385556 12562 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:13.385717 12562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:13.392045 12562 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:08:13.395709 12577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.397435 12562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:13.397691 12562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "bc75f05864744604a399396a8f14e356"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "52efdd3f053fcd73281502eab7fbd66a"
server_key_iv: "2cbf8a718918ba234a5fc71702a037d2"
server_key_version: "encryptionkey@0"
I20260806 09:08:13.398917 12562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:13.425402 12562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:13.430228 12562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:13.430539 12562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:13.431655 12562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:13.433100 12562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:13.433235 12562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.433365 12562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:13.433452 12562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.469897 12562 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38385
I20260806 09:08:13.469949 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38385 every 8 connection(s)
I20260806 09:08:13.471625 12562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:08:13.474378 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12562
I20260806 09:08:13.475265 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:45025
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=36537
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:08:13 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:13.499018 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:13.475855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:50917 (local address 127.27.187.190:40813)
0806 09:08:13.476108 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:13.476115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:13.477356 (+ 1241us) server_negotiation.cc:407] Connection header received
0806 09:08:13.479811 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:13.479818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:13.479939 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:13.480059 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:13.482131 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.482975 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.484741 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.485229 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.485392 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:13.489924 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:13.489934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:13.489942 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:13.489999 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:13.494399 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:13.495678 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:13.495687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:13.495693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:13.495877 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:13.496553 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:13.496561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:13.496567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:13.497015 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:13.497195 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:13.498173 (+ 978us) server_negotiation.cc:299] Negotiation successful
0806 09:08:13.498515 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:08:13.499825 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:13.476071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:50917)
0806 09:08:13.477071 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:13.477125 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:13.479435 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:13.480413 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:13.480424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:13.481149 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:13.481877 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:13.481901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.483184 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.483189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:13.484544 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:13.484560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.485521 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.485529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:13.485797 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:13.486624 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:13.486650 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:13.489524 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:13.494658 (+ 5134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:13.494670 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:13.494756 (+ 86us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:13.495496 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:13.496048 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:13.496052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:13.496055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:13.496394 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:13.497229 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:13.497237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:13.497698 (+ 461us) client_negotiation.cc:769] Sending connection context
0806 09:08:13.498016 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 09:08:13.498495 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":411,"threads_started":1}
I20260806 09:08:13.501701 12691 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:08:13.502035 12691 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:13.502895 12691 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:08:13.504666 12371 ts_manager.cc:194] Registered new tserver with Master: bc75f05864744604a399396a8f14e356 (127.27.187.130:38385)
I20260806 09:08:13.506073 12371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:50917
W20260806 09:08:13.851979 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:13.852337 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:13.852540 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:13.860370 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:13.860550 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:13.860764 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:08:13.871591 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:45025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=36537
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:13.873102 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:13.874743 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:13.889511 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:13.890014 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:13.890051 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:13.891099 12694 server_base.cc:1034] running on GCE node
I20260806 09:08:13.891902 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:13.892946 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:13.894253 12694 hybrid_clock.cc:648] HybridClock initialized: now 1786007293894123 us; error 36 us; skew 500 ppm
I20260806 09:08:13.894731 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:13 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:13.898367 12694 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:08:13.900256 12694 webserver.cc:460] Webserver started at http://127.27.187.131:36537/ using document root <none> and password file <none>
I20260806 09:08:13.901033 12694 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:13.901213 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:13.907653 12694 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:08:13.913017 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.915021 12694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:08:13.915261 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "9c9e4397cb213b68bfd6087b101512b8"
server_key_iv: "2e417393127c8171eb70d05e2625dfe5"
server_key_version: "encryptionkey@0"
I20260806 09:08:13.916388 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:13.935483 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:13.940591 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:13.940912 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:13.942191 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:13.943825 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:13.943961 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.944109 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:13.944222 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:13.982573 12694 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45025
I20260806 09:08:13.982735 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45025 every 8 connection(s)
I20260806 09:08:13.984189 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:08:13.994266 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12694
I20260806 09:08:13.997483 12558 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
Aug 06 09:08:13 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:14.010991 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:13.987322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:42709 (local address 127.27.187.190:40813)
0806 09:08:13.987548 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:13.987553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:13.988879 (+ 1326us) server_negotiation.cc:407] Connection header received
0806 09:08:13.991228 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:13.991234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:13.991343 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:13.991468 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:13.993327 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.994503 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.997358 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.998216 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.998401 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.003693 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.003699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.003703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.003748 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.007295 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.008243 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.008248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.008253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.008401 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.008932 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.008936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.008938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.009248 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.009467 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.010252 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.010523 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:08:14.012485 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:13.987800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:42709)
0806 09:08:13.988678 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:13.988730 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:13.990847 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:13.991767 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:13.991777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:13.992424 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:13.993091 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:13.993111 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.994787 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.994792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:13.997135 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:13.997150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:13.998490 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:13.998499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:13.998867 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:13.999939 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:13.999973 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:14.003391 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:14.007505 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:14.007516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:14.007546 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:14.008078 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:14.008538 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:14.008545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:14.008548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:14.008784 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:14.009408 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:14.009417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:14.009857 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.010143 (+ 286us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.010824 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":341,"threads_started":1}
I20260806 09:08:14.014264 12823 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40813
I20260806 09:08:14.014629 12823 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:14.015422 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:13.995528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.27.187.190:40813)
0806 09:08:13.996508 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:13.996514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:13.996535 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:08:13.996680 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:13.996686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:13.996771 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:13.996892 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:13.999597 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.000834 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.003418 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.004110 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.004297 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.007920 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.007926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.007930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.007970 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.011272 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.012442 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.012447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.012450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.012733 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.013586 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.013592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.013595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.013958 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.014434 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.014775 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.015038 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":664,"threads_started":1}
I20260806 09:08:14.015606 12823 heartbeater.cc:511] Master 127.27.187.190:40813 requested a full tablet report, sending...
I20260806 09:08:14.018004 12371 ts_manager.cc:194] Registered new tserver with Master: 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
I20260806 09:08:14.018998 12371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:42709
I20260806 09:08:14.031417 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:08:14.038591 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.032482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.27.187.190:40813)
0806 09:08:14.032707 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.032712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.033189 (+ 477us) server_negotiation.cc:407] Connection header received
0806 09:08:14.033413 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.033417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.033572 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.033710 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:08:14.034153 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.034977 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.036478 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.036948 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.037055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.037669 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:08:14.037967 (+ 298us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 09:08:14.038354 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47268: token expired
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 09:08:14.038821 12827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47268: token expired
W20260806 09:08:14.039465 12294 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.187.190:40813)
I20260806 09:08:14.055263 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.040069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.27.187.190:40813)
0806 09:08:14.040312 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.040316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.040340 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:08:14.040546 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.040549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.040633 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.040739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:14.042284 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.043137 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.044878 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.045339 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.045467 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.048874 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.048879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.048883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.048924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.051945 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.052858 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.052868 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.052870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.053037 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.053566 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.053570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.053572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.053988 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.054209 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.054567 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.054847 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:08:14.059202 12371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:08:14.067106 12371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:08:14.094905 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.081340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:51286)
0806 09:08:14.083330 (+ 1990us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.083378 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:14.083583 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:14.084911 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:14.084918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:14.084954 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:14.085600 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.085612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.087478 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.087483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.090145 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.090160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.093266 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.093276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.093527 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.093895 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.094072 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.094307 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":282,"thread_start_us":5070,"threads_started":1}
I20260806 09:08:14.095086 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.081697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.27.187.129:43255)
0806 09:08:14.082762 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.082773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.083784 (+ 1011us) server_negotiation.cc:407] Connection header received
0806 09:08:14.083894 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.083903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.084178 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.084614 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:14.085794 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.087305 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.090329 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.091956 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.093680 (+ 1724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.093897 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.094512 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.094735 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":1000,"threads_started":1}
I20260806 09:08:14.100025 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.081821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45025 (local address 127.0.0.1:60096)
0806 09:08:14.086715 (+ 4894us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.086738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:14.086865 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:14.087943 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:14.087948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:14.087974 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:14.090608 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.090620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.092837 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.092842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.097050 (+ 4208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.097069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.099159 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.099166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.099334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.099374 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.099471 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.099614 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4732,"thread_start_us":1456,"threads_started":1}
I20260806 09:08:14.100198 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.082468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.27.187.131:45025)
0806 09:08:14.086327 (+ 3859us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.086337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.087025 (+ 688us) server_negotiation.cc:407] Connection header received
0806 09:08:14.087127 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.087134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.087412 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.087610 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:14.090872 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.092659 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.097275 (+ 4616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.098881 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.099111 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.099300 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.099638 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.099836 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3674,"thread_start_us":3493,"threads_started":1}
I20260806 09:08:14.101171 12492 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0376f858254005915c2e2728de4cc3 (DEFAULT_TABLE table=test-table [id=8deb8832856740f38950c22e041596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:14.103142 12492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0376f858254005915c2e2728de4cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:14.104166 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.087209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.27.187.130:38385)
0806 09:08:14.096616 (+ 9407us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.096626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.096658 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:08:14.096795 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.096807 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.097160 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.097398 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:14.098050 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.099592 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.101868 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.103063 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.103245 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.103443 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.103685 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.103847 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9255,"thread_start_us":9138,"threads_started":1}
I20260806 09:08:14.104204 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.086902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38385 (local address 127.0.0.1:52450)
0806 09:08:14.095730 (+ 8828us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.095746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:14.095873 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:14.097659 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:14.097664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:14.097694 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:14.097871 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.097883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.099766 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.099773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.101648 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.101667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.103248 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.103255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.103432 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.103473 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.103602 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.103767 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8743,"thread_start_us":9708,"threads_started":1}
I20260806 09:08:14.104470 12758 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0376f858254005915c2e2728de4cc3 (DEFAULT_TABLE table=test-table [id=8deb8832856740f38950c22e041596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:14.106072 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0376f858254005915c2e2728de4cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:14.107695 12626 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0376f858254005915c2e2728de4cc3 (DEFAULT_TABLE table=test-table [id=8deb8832856740f38950c22e041596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:14.109122 12626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0376f858254005915c2e2728de4cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:14.121598 12839 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Bootstrap starting.
I20260806 09:08:14.123062 12840 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Bootstrap starting.
I20260806 09:08:14.126634 12839 tablet_bootstrap.cc:654] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:14.127616 12841 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Bootstrap starting.
I20260806 09:08:14.127960 12840 tablet_bootstrap.cc:654] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:14.128288 12839 log.cc:824] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:14.129431 12840 log.cc:824] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:14.131577 12841 tablet_bootstrap.cc:654] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:14.132359 12839 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: No bootstrap required, opened a new log
I20260806 09:08:14.132678 12839 ts_tablet_manager.cc:1397] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260806 09:08:14.132997 12841 log.cc:824] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:14.133494 12840 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: No bootstrap required, opened a new log
I20260806 09:08:14.133836 12840 ts_tablet_manager.cc:1397] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260806 09:08:14.136701 12841 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: No bootstrap required, opened a new log
I20260806 09:08:14.137017 12841 ts_tablet_manager.cc:1397] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:08:14.141742 12839 raft_consensus.cc:348] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.142227 12839 raft_consensus.cc:374] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:14.142366 12839 raft_consensus.cc:729] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70018f3b149240398798fdcddd320b81, State: Initialized, Role: FOLLOWER
I20260806 09:08:14.143102 12839 consensus_queue.cc:260] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.144905 12839 ts_tablet_manager.cc:1428] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260806 09:08:14.146126 12840 raft_consensus.cc:348] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.146646 12840 raft_consensus.cc:374] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:14.146797 12840 raft_consensus.cc:729] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc75f05864744604a399396a8f14e356, State: Initialized, Role: FOLLOWER
I20260806 09:08:14.147477 12840 consensus_queue.cc:260] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.147972 12841 raft_consensus.cc:348] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.148535 12841 raft_consensus.cc:374] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:14.148689 12841 raft_consensus.cc:729] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fc57e5ef6041dcb11dd6fe5aedc2cc, State: Initialized, Role: FOLLOWER
I20260806 09:08:14.148938 12691 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
I20260806 09:08:14.149650 12840 ts_tablet_manager.cc:1428] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260806 09:08:14.149498 12841 consensus_queue.cc:260] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.151582 12823 heartbeater.cc:503] Master 127.27.187.190:40813 was elected leader, sending a full tablet report...
I20260806 09:08:14.152472 12841 ts_tablet_manager.cc:1428] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
W20260806 09:08:14.209746 12559 tablet.cc:2369] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:08:14.215109 12847 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:14.215492 12847 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.217506 12847 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255), bc75f05864744604a399396a8f14e356 (127.27.187.130:38385)
W20260806 09:08:14.224145 12692 tablet.cc:2369] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:08:14.225792 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.218108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:45565 (local address 127.27.187.129:43255)
0806 09:08:14.218350 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.218358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.218382 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:08:14.218737 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.218744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.218829 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.218931 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:14.219734 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.220803 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.223249 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.224527 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.224723 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.224804 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.225346 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.225498 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:08:14.225793 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.217975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.27.187.131:45565)
0806 09:08:14.218268 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.218293 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:14.218471 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:14.219155 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:14.219160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:14.219295 (+ 135us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:14.219552 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.219566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.220949 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.220953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.223011 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.223029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.224914 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.224923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.225133 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.225174 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.225300 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.225454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":4}
I20260806 09:08:14.226661 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.218508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38385 (local address 127.27.187.131:44633)
0806 09:08:14.219421 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.219442 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:14.219578 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:14.220082 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:14.220086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:14.220113 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:14.220502 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.220512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.221818 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.221824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.223641 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:14.223653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.225776 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.225788 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:14.226113 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.226154 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:08:14.226240 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 09:08:14.226353 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":472,"threads_started":1}
I20260806 09:08:14.226720 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.218784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:44633 (local address 127.27.187.130:38385)
0806 09:08:14.219056 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.219063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.219574 (+ 511us) server_negotiation.cc:407] Connection header received
0806 09:08:14.219728 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.219734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.219827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.219971 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:14.220685 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.221667 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.223829 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.225455 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.225640 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.225707 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.226306 (+ 599us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.226449 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:08:14.227169 12512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0376f858254005915c2e2728de4cc3" candidate_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70018f3b149240398798fdcddd320b81" is_pre_election: true
I20260806 09:08:14.227599 12646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0376f858254005915c2e2728de4cc3" candidate_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc75f05864744604a399396a8f14e356" is_pre_election: true
I20260806 09:08:14.227712 12512 raft_consensus.cc:2454] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fc57e5ef6041dcb11dd6fe5aedc2cc in term 0.
I20260806 09:08:14.228051 12646 raft_consensus.cc:2454] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83fc57e5ef6041dcb11dd6fe5aedc2cc in term 0.
I20260806 09:08:14.228482 12712 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70018f3b149240398798fdcddd320b81, 83fc57e5ef6041dcb11dd6fe5aedc2cc; no voters:
I20260806 09:08:14.228919 12847 raft_consensus.cc:2781] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:08:14.229074 12847 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:08:14.229204 12847 raft_consensus.cc:3037] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:14.231664 12847 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.232383 12847 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [CANDIDATE]: Term 1 election: Requested vote from peers 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255), bc75f05864744604a399396a8f14e356 (127.27.187.130:38385)
I20260806 09:08:14.233017 12512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0376f858254005915c2e2728de4cc3" candidate_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70018f3b149240398798fdcddd320b81"
I20260806 09:08:14.233126 12646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0376f858254005915c2e2728de4cc3" candidate_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc75f05864744604a399396a8f14e356"
I20260806 09:08:14.233250 12512 raft_consensus.cc:3037] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:14.233415 12646 raft_consensus.cc:3037] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:14.235805 12512 raft_consensus.cc:2454] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fc57e5ef6041dcb11dd6fe5aedc2cc in term 1.
I20260806 09:08:14.235858 12646 raft_consensus.cc:2454] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83fc57e5ef6041dcb11dd6fe5aedc2cc in term 1.
W20260806 09:08:14.235975 12824 tablet.cc:2369] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:08:14.236344 12712 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356; no voters:
I20260806 09:08:14.236778 12847 raft_consensus.cc:2781] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:08:14.237368 12847 raft_consensus.cc:686] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 LEADER]: Becoming Leader. State: Replica: 83fc57e5ef6041dcb11dd6fe5aedc2cc, State: Running, Role: LEADER
I20260806 09:08:14.237890 12847 consensus_queue.cc:237] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:14.243461 12370 catalog_manager.cc:5495] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131). New cstate: current_term: 1 leader_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } health_report { overall_health: HEALTHY } } }
I20260806 09:08:14.293058 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.287300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.27.187.131:45025)
0806 09:08:14.287529 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.287536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.287558 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:08:14.287797 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.287801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.287887 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.288054 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:08:14.288477 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.289224 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.290715 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.291137 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.291283 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.291850 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:08:14.292420 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:08:14.292601 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.292685 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.292777 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:08:14.302634 12512 raft_consensus.cc:1261] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Refusing update from remote peer 83fc57e5ef6041dcb11dd6fe5aedc2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:08:14.302596 12646 raft_consensus.cc:1261] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Refusing update from remote peer 83fc57e5ef6041dcb11dd6fe5aedc2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:08:14.303885 12849 consensus_queue.cc:1035] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:08:14.304291 12847 consensus_queue.cc:1035] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:08:14.314888 12852 mvcc.cc:205] Tried to move back new op lower bound from 7315485877451161600 to 7315485877203021824. Current Snapshot: MvccSnapshot[applied={T|T < 7315485877451161600}]
I20260806 09:08:14.316309 12854 mvcc.cc:205] Tried to move back new op lower bound from 7315485877451161600 to 7315485877203021824. Current Snapshot: MvccSnapshot[applied={T|T < 7315485877451161600}]
I20260806 09:08:14.333830 12371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:08:14.334173 12371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:08:14.337869 12371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Sending DeleteTablet for 3 replicas of tablet 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:14.339298 12758 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0376f858254005915c2e2728de4cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:14 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:60096
I20260806 09:08:14.339313 12492 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0376f858254005915c2e2728de4cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:14 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:51286
I20260806 09:08:14.339502 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12408
I20260806 09:08:14.340377 12626 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a0376f858254005915c2e2728de4cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:14 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:52450
I20260806 09:08:14.365898 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.341129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.27.187.129:43255)
0806 09:08:14.341342 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.341350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.341372 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:08:14.341442 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.341450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.341556 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.341667 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:14.344185 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.345669 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.348209 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.349125 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.349276 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.354280 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.354307 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.354336 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.354400 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.359662 (+ 5262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.360985 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.361000 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.361007 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.361255 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.362279 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.362287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.362293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.362994 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.363368 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.364985 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.365314 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:08:14.367354 12553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:14.375726 12863 tablet_replica.cc:330] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: stopping tablet replica
I20260806 09:08:14.376863 12863 raft_consensus.cc:2229] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:08:14.378099 12861 tablet_replica.cc:330] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: stopping tablet replica
I20260806 09:08:14.379837 12863 raft_consensus.cc:2258] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:08:14.380514 12861 raft_consensus.cc:2229] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:08:14.382241 12861 raft_consensus.cc:2258] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:08:14.384863 12863 ts_tablet_manager.cc:1905] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:08:14.388101 12861 ts_tablet_manager.cc:1905] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:08:14.402486 12863 ts_tablet_manager.cc:1918] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:08:14.402994 12863 log.cc:1196] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/8a0376f858254005915c2e2728de4cc3
I20260806 09:08:14.403236 12861 ts_tablet_manager.cc:1918] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:08:14.403534 12861 log.cc:1196] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/8a0376f858254005915c2e2728de4cc3
I20260806 09:08:14.403848 12863 ts_tablet_manager.cc:1939] T 8a0376f858254005915c2e2728de4cc3 P bc75f05864744604a399396a8f14e356: Deleting consensus metadata
I20260806 09:08:14.404201 12861 ts_tablet_manager.cc:1939] T 8a0376f858254005915c2e2728de4cc3 P 83fc57e5ef6041dcb11dd6fe5aedc2cc: Deleting consensus metadata
I20260806 09:08:14.406293 12358 catalog_manager.cc:4841] TS 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): tablet 8a0376f858254005915c2e2728de4cc3 (table test-table [id=8deb8832856740f38950c22e041596c4]) successfully deleted
I20260806 09:08:14.406692 12357 catalog_manager.cc:4841] TS bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): tablet 8a0376f858254005915c2e2728de4cc3 (table test-table [id=8deb8832856740f38950c22e041596c4]) successfully deleted
I20260806 09:08:14.512668 12862 tablet_replica.cc:330] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: stopping tablet replica
I20260806 09:08:14.513075 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12408
W20260806 09:08:14.539144 12357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:43255 (error 108)
I20260806 09:08:14.540071 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12562
I20260806 09:08:14.541133 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.540613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59458)
0806 09:08:14.540820 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.540937 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:08:14.541541 12357 catalog_manager.cc:4584] TS 70018f3b149240398798fdcddd320b81 (127.27.187.129:43255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a0376f858254005915c2e2728de4cc3: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
I20260806 09:08:14.558547 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.541330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.27.187.130:38385)
0806 09:08:14.541547 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.541555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.541627 (+ 72us) server_negotiation.cc:407] Connection header received
0806 09:08:14.541696 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.541703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.541791 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.541893 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:14.543643 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.544990 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.546569 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.547382 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.547540 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.550586 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.550610 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.550634 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.550729 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.555060 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.556044 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.556053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.556056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.556251 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.556678 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.556684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.556687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.557110 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.557555 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.557905 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.558164 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:08:14.559356 12686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:14.595453 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.594866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59464)
0806 09:08:14.595130 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.595269 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 09:08:14.597015 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.596504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59466)
0806 09:08:14.596698 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.596815 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:08:14.659044 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.657536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59470)
0806 09:08:14.658723 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.658845 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":981}
I20260806 09:08:14.659192 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12562
I20260806 09:08:14.662987 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.662412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59474)
0806 09:08:14.662630 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.662763 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 09:08:14.678964 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12694
I20260806 09:08:14.698853 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.679923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.27.187.131:45025)
0806 09:08:14.680120 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:14.680124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:14.680248 (+ 124us) server_negotiation.cc:407] Connection header received
0806 09:08:14.680454 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:14.680459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:14.680540 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:14.680660 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:14.682800 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.684038 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.686261 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:14.686844 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:14.686965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:14.691105 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:14.691130 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:14.691157 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:14.691213 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:14.695364 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.696256 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.696268 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.696272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.696439 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:14.696882 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:14.696888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:14.696896 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:14.697320 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:14.697719 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:14.698055 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 09:08:14.698354 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:08:14.699767 12818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:14.744383 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.743769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59478)
0806 09:08:14.744043 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.744180 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 09:08:14.745880 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.745321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59484)
0806 09:08:14.745580 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.745684 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 09:08:14.842108 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12694
I20260806 09:08:14.861581 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12338
I20260806 09:08:14.863029 12401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:14.984282 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:14.983726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43255 (local address 127.0.0.1:59500)
0806 09:08:14.983969 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:14.984078 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:08:14.984445 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12338
I20260806 09:08:35.013576 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--webserver_interface=127.27.187.190
--webserver_port=36875
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:08:35.384550 12869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:35.384972 12869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:35.385164 12869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:35.392903 12869 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:08:35.393105 12869 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:08:35.393203 12869 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:08:35.393299 12869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:08:35.393386 12869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:35.393488 12869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:35.393577 12869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:08:35.393659 12869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:08:35.404449 12869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=36875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:35.405977 12869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:35.407554 12869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:35.422214 12874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:35.422312 12875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:35.422333 12869 server_base.cc:1034] running on GCE node
W20260806 09:08:35.422226 12877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:35.423774 12869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:35.425608 12869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:35.426908 12869 hybrid_clock.cc:648] HybridClock initialized: now 1786007315426821 us; error 57 us; skew 500 ppm
I20260806 09:08:35.427409 12869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:35 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:35.439527 12869 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:08:35.441583 12869 webserver.cc:460] Webserver started at http://127.27.187.190:36875/ using document root <none> and password file <none>
I20260806 09:08:35.442474 12869 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:35.442641 12869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:35.449517 12869 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 09:08:35.453213 12884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:35.455086 12869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:08:35.455333 12869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "77180c5858cd4bb99fd6a351c4f2faa8"
format_stamp: "Formatted at 2026-08-06 09:07:48 on dist-test-slave-8tdl"
server_key: "3117e21f48b7fb05b444c37f549d6392"
server_key_iv: "93cf6ac2869c85be77e64e19c5b7d094"
server_key_version: "encryptionkey@0"
I20260806 09:08:35.456461 12869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:35.477599 12869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:35.483474 12869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:35.483855 12869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:35.508026 12869 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:40813
I20260806 09:08:35.508067 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:40813 every 8 connection(s)
I20260806 09:08:35.509791 12869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:08:35.518478 12936 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:08:35.519634 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12869
I20260806 09:08:35.520606 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:43255
--local_ip_for_outbound_sockets=127.27.187.129
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=37659
--webserver_interface=127.27.187.129
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:08:35.522374 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap starting.
I20260806 09:08:35.546518 12936 log.cc:824] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:35.573751 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:08:35.574602 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Bootstrap complete.
I20260806 09:08:35.591755 12936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:35.593577 12936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Initialized, Role: FOLLOWER
I20260806 09:08:35.594782 12936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:35.595187 12936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:08:35.595350 12936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:08:35.595726 12936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:08:35.603549 12936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:35.604357 12936 leader_election.cc:302] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77180c5858cd4bb99fd6a351c4f2faa8; no voters:
I20260806 09:08:35.605233 12936 leader_election.cc:288] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:08:35.605410 12940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:08:35.607752 12936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:08:35.612983 12940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [term 3 LEADER]: Becoming Leader. State: Replica: 77180c5858cd4bb99fd6a351c4f2faa8, State: Running, Role: LEADER
I20260806 09:08:35.613868 12940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } }
I20260806 09:08:35.622313 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:08:35.622769 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:35.628319 12953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:08:35.628901 12942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77180c5858cd4bb99fd6a351c4f2faa8. Latest consensus state: current_term: 3 leader_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77180c5858cd4bb99fd6a351c4f2faa8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40813 } } }
I20260806 09:08:35.629550 12942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:35.638630 12953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:08:35.640734 12953 catalog_manager.cc:1259] Loaded cluster ID: 28318c1e83bc4314842c794feb290c70
I20260806 09:08:35.640985 12953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:08:35.645751 12953 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:08:35.647701 12953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Loaded TSK: 0
I20260806 09:08:35.648003 12953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77180c5858cd4bb99fd6a351c4f2faa8: Loaded TSK: 1
F20260806 09:08:35.649223 12953 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc6871bc9ad google::LogMessage::Fail() at ??:0
@ 0x7fc6871bf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc6871bc4de google::LogMessage::Flush() at ??:0
@ 0x7fc6871bda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc68af207d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc68af0f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc69cbeb1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc69cbea783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc69cc762a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fc69cc760c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc69ccc05e9 std::function<>::operator()() at ??:0
@ 0x7fc69cbeecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc69cbecc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc69cc74610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc69cc74439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc689e08345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc689e0b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc689e0b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc689de6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc68d0c06db start_thread at ??:0
@ 0x7fc68605661f clone at ??:0
W20260806 09:08:36.015976 12938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:36.016332 12938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:36.016515 12938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:36.024477 12938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:36.024638 12938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:36.024832 12938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:08:36.035781 12938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:43255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=37659
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:36.037314 12938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:36.038939 12938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:36.054209 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:36.054284 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:36.054243 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:36.054824 12938 server_base.cc:1034] running on GCE node
I20260806 09:08:36.055653 12938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:36.056805 12938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:36.058107 12938 hybrid_clock.cc:648] HybridClock initialized: now 1786007316058035 us; error 52 us; skew 500 ppm
I20260806 09:08:36.058531 12938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:36 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:36.062297 12938 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:08:36.064299 12938 webserver.cc:460] Webserver started at http://127.27.187.129:37659/ using document root <none> and password file <none>
I20260806 09:08:36.065109 12938 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:36.065274 12938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:36.071818 12938 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:08:36.075519 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:36.077476 12938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:36.077765 12938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "70018f3b149240398798fdcddd320b81"
format_stamp: "Formatted at 2026-08-06 09:07:49 on dist-test-slave-8tdl"
server_key: "2b4aaa579fdc1d14bbb2cf4846451abe"
server_key_iv: "3f454cf7d9d1400b9e1ac3c26e45ba79"
server_key_version: "encryptionkey@0"
I20260806 09:08:36.079442 12938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:36.097465 12938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:36.103286 12938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:36.103655 12938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:36.104908 12938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:36.107491 13001 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 09:08:36.114555 12938 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 09:08:36.114830 12938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260806 09:08:36.115010 12938 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 09:08:36.118853 12938 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 09:08:36.119030 12938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.002s
I20260806 09:08:36.119424 13001 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Bootstrap starting.
I20260806 09:08:36.150352 13001 log.cc:824] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:36.167038 13001 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:08:36.167894 13001 tablet_bootstrap.cc:492] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Bootstrap complete.
I20260806 09:08:36.168448 12938 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43255
I20260806 09:08:36.168606 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43255 every 8 connection(s)
I20260806 09:08:36.169713 13001 ts_tablet_manager.cc:1397] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Time spent bootstrapping tablet: real 0.051s user 0.027s sys 0.020s
I20260806 09:08:36.169946 12938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:08:36.171329 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 12938
I20260806 09:08:36.172215 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:38385
--local_ip_for_outbound_sockets=127.27.187.130
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=35927
--webserver_interface=127.27.187.130
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:08:36.182951 13001 raft_consensus.cc:348] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:36.184916 13001 raft_consensus.cc:729] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70018f3b149240398798fdcddd320b81, State: Initialized, Role: FOLLOWER
I20260806 09:08:36.185578 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.174583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:54691)
0806 09:08:36.183660 (+ 9077us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.184084 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8653,"thread_start_us":8217,"threads_started":1}
I20260806 09:08:36.185792 13001 consensus_queue.cc:260] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
W20260806 09:08:36.187001 13112 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:40813 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:40813: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
I20260806 09:08:36.188184 13001 ts_tablet_manager.cc:1428] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260806 09:08:36.188319 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.187715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:37271)
0806 09:08:36.187940 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.188053 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:08:36.189603 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.189106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:50435)
0806 09:08:36.189339 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.189445 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 09:08:36.190145 13112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:08:36.555085 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:36.555480 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:36.555665 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:36.563485 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:36.563673 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:36.563870 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:08:36.574285 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:38385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=35927
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:36.575747 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:36.577229 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:36.592734 13122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:36.592831 13124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:36.592856 13121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:36.593649 13115 server_base.cc:1034] running on GCE node
I20260806 09:08:36.594486 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:36.595723 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:36.596971 13115 hybrid_clock.cc:648] HybridClock initialized: now 1786007316596920 us; error 46 us; skew 500 ppm
I20260806 09:08:36.597311 13115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:36 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:36.601136 13115 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:08:36.602917 13115 webserver.cc:460] Webserver started at http://127.27.187.130:35927/ using document root <none> and password file <none>
I20260806 09:08:36.603655 13115 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:36.603771 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:36.610306 13115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 09:08:36.613792 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:36.615563 13115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:08:36.615823 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "bc75f05864744604a399396a8f14e356"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "52efdd3f053fcd73281502eab7fbd66a"
server_key_iv: "2cbf8a718918ba234a5fc71702a037d2"
server_key_version: "encryptionkey@0"
I20260806 09:08:36.616977 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:36.632789 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:36.638283 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:36.638664 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:36.640038 13115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:36.641754 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:36.641871 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:36.642047 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:36.642167 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:36.682116 13115 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38385
I20260806 09:08:36.682332 13244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38385 every 8 connection(s)
I20260806 09:08:36.683861 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:08:36.689565 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.687454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:48653)
0806 09:08:36.688300 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.688574 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":381,"threads_started":1}
W20260806 09:08:36.690572 13245 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:40813 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:40813: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
I20260806 09:08:36.691705 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13115
I20260806 09:08:36.691943 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.691490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:52467)
0806 09:08:36.691691 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.691772 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:08:36.692574 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:45025
--local_ip_for_outbound_sockets=127.27.187.131
--tserver_master_addrs=127.27.187.190:40813
--webserver_port=36537
--webserver_interface=127.27.187.131
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:08:36.693112 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:36.692575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:48813)
0806 09:08:36.692804 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:36.692926 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 09:08:36.693444 13245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:08:37.061079 13248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:37.061411 13248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:37.061584 13248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:37.069371 13248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:37.069516 13248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:37.069701 13248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:08:37.080003 13248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:45025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=36537
--tserver_master_addrs=127.27.187.190:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:37.081501 13248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:37.082986 13248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:37.098829 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:37.098946 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:37.098904 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:37.100270 13248 server_base.cc:1034] running on GCE node
I20260806 09:08:37.101105 13248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:37.102358 13248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:37.103637 13248 hybrid_clock.cc:648] HybridClock initialized: now 1786007317103510 us; error 115 us; skew 500 ppm
I20260806 09:08:37.104064 13248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:37 dist-test-slave-8tdl krb5kdc[11779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007317, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:37.107851 13248 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:08:37.109972 13248 webserver.cc:460] Webserver started at http://127.27.187.131:36537/ using document root <none> and password file <none>
I20260806 09:08:37.110833 13248 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:37.110987 13248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:37.117203 13248 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:08:37.120566 13263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:37.122731 13248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 09:08:37.122997 13248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc"
format_stamp: "Formatted at 2026-08-06 09:07:50 on dist-test-slave-8tdl"
server_key: "9c9e4397cb213b68bfd6087b101512b8"
server_key_iv: "2e417393127c8171eb70d05e2625dfe5"
server_key_version: "encryptionkey@0"
I20260806 09:08:37.124132 13248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:37.144910 13248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:37.150010 13248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:37.150425 13248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:37.151607 13248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:37.153227 13248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:37.153321 13248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:37.153482 13248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:37.153563 13248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:37.192104 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.190969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:43961)
0806 09:08:37.191637 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.191772 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":275,"threads_started":1}
I20260806 09:08:37.192237 13248 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:45025
I20260806 09:08:37.192404 13379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:45025 every 8 connection(s)
I20260806 09:08:37.193823 13248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:08:37.199640 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13248
I20260806 09:08:37.199646 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.197451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:47621)
0806 09:08:37.198293 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.198574 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
W20260806 09:08:37.200770 13380 heartbeater.cc:644] Failed to heartbeat to 127.27.187.190:40813 (0 consecutive failures): Network error: Failed to ping master at 127.27.187.190:40813: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
I20260806 09:08:37.202078 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.201622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:60121)
0806 09:08:37.201828 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.201906 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 09:08:37.203182 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.202812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:53025)
0806 09:08:37.202967 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.203052 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 09:08:37.203589 13380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:08:37.642650 13384 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:37.643087 13384 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:37.644994 13384 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
Aug 06 09:08:37 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
Aug 06 09:08:37 dist-test-slave-8tdl krb5kdc[11779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:08:37.673519 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.645963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:55761 (local address 127.27.187.131:45025)
0806 09:08:37.648234 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:37.648243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:37.648278 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:08:37.649811 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:37.649821 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:37.650258 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:37.650599 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:37.653080 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.654928 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.657193 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.658036 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.658408 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:37.663514 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:37.663524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:37.663542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:37.663596 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:37.668911 (+ 5315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670144 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:37.670155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:37.670170 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:37.670365 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670865 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:37.670874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:37.670876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:37.671514 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:37.671734 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:37.672631 (+ 897us) server_negotiation.cc:299] Negotiation successful
0806 09:08:37.673045 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094,"thread_start_us":1966,"threads_started":1}
I20260806 09:08:37.673525 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.645514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:45025 (local address 127.27.187.129:55761)
0806 09:08:37.645876 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.645919 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:37.646728 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:37.650861 (+ 4133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:37.650873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:37.651782 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:37.652786 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:37.652812 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.655208 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.655216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:37.656979 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:37.657001 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.658325 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.658332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:37.658655 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:37.659661 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:37.659696 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:37.663079 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:37.669184 (+ 6105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:37.669194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:37.669213 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:37.669895 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:37.670505 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:37.670516 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:37.670729 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:37.671690 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:37.671695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:37.672066 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 09:08:37.672370 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 09:08:37.673022 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":218,"spinlock_wait_cycles":15488}
I20260806 09:08:37.673624 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.645987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:41263 (local address 127.27.187.130:38385)
0806 09:08:37.647743 (+ 1756us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:37.647767 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:37.648210 (+ 443us) server_negotiation.cc:407] Connection header received
0806 09:08:37.649814 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:37.649823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:37.650265 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:37.650614 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:37.653070 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.654990 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.657201 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.658015 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.658255 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:37.663846 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:37.663856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:37.663870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:37.663918 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:37.668908 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670068 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:37.670077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:37.670084 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:37.670277 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670848 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:37.670852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:37.670854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:37.671520 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:37.671739 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:37.672623 (+ 884us) server_negotiation.cc:299] Negotiation successful
0806 09:08:37.673035 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":1331,"threads_started":1}
I20260806 09:08:37.674336 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.645599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38385 (local address 127.27.187.129:41263)
0806 09:08:37.648068 (+ 2469us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.648086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:37.648251 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:37.650862 (+ 2611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:37.650873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:37.651826 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:37.652783 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:37.652811 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.655325 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.655329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:37.656980 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:37.657001 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:37.658276 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:37.658283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:37.658656 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:37.659696 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:37.659724 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:37.663493 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:37.669182 (+ 5689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:37.669195 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:37.669211 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:37.669867 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:37.670421 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:37.670427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:37.670430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:37.670651 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:37.671678 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:37.671687 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:37.672066 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 09:08:37.672370 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 09:08:37.673433 (+ 1063us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2322,"mutex_wait_us":531,"spinlock_wait_cycles":86784,"thread_start_us":2120,"threads_started":1}
W20260806 09:08:37.677708 12998 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
W20260806 09:08:37.678043 12997 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:37.678243 12997 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70018f3b149240398798fdcddd320b81; no voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356
I20260806 09:08:37.678855 13384 raft_consensus.cc:2726] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:08:37.695600 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:37.694366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:42451)
0806 09:08:37.694984 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:37.695244 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":337,"threads_started":1}
I20260806 09:08:38.194099 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:38.193234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:47429)
0806 09:08:38.193738 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:38.193916 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":254,"threads_started":1}
I20260806 09:08:38.205391 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:38.204341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:42973)
0806 09:08:38.205023 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:38.205174 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":409,"threads_started":1}
I20260806 09:08:38.697842 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:38.696826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:47935)
0806 09:08:38.697543 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:38.697656 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":274,"threads_started":1}
I20260806 09:08:39.196535 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:39.195441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:36123)
0806 09:08:39.196108 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:39.196239 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":339,"threads_started":1}
I20260806 09:08:39.207552 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:39.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:33691)
0806 09:08:39.207193 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:39.207307 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":367,"threads_started":1}
I20260806 09:08:39.257431 13394 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:39.257711 13394 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:39.258574 13394 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
W20260806 09:08:39.259609 12998 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
W20260806 09:08:39.259838 12997 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:39.259928 12997 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70018f3b149240398798fdcddd320b81; no voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356
I20260806 09:08:39.260260 13394 raft_consensus.cc:2726] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:08:39.699852 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:39.698982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:33565)
0806 09:08:39.699479 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:39.699576 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260806 09:08:40.198618 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:40.197699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:47057)
0806 09:08:40.198331 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:40.198431 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":349,"threads_started":1}
I20260806 09:08:40.209611 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:40.208683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:44955)
0806 09:08:40.209304 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:40.209418 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":315,"threads_started":1}
I20260806 09:08:40.701609 13398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:40.700858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:45139)
0806 09:08:40.701321 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:40.701413 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260806 09:08:41.200868 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:41.199944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:42279)
0806 09:08:41.200570 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:41.200661 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":358,"threads_started":1}
I20260806 09:08:41.211735 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:41.210802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:35791)
0806 09:08:41.211421 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:41.211524 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":325,"threads_started":1}
I20260806 09:08:41.703627 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:41.702762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:59803)
0806 09:08:41.703331 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:41.703441 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":223,"threads_started":1}
I20260806 09:08:42.074193 13402 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:42.074487 13402 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:42.075400 13402 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
W20260806 09:08:42.076668 12998 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
W20260806 09:08:42.076889 12997 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:42.077054 12997 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70018f3b149240398798fdcddd320b81; no voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356
I20260806 09:08:42.077364 13402 raft_consensus.cc:2726] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:08:42.203229 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:42.202079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:50847)
0806 09:08:42.202807 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:42.202977 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260806 09:08:42.213840 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:42.212895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:34379)
0806 09:08:42.213486 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:42.213595 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":330,"threads_started":1}
I20260806 09:08:42.705783 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:42.704849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:51109)
0806 09:08:42.705460 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:42.705558 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":348,"threads_started":1}
I20260806 09:08:43.205092 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:43.204260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:60715)
0806 09:08:43.204765 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:43.204862 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":259,"threads_started":1}
I20260806 09:08:43.215953 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:43.214956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:48931)
0806 09:08:43.215643 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:43.215743 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":393,"threads_started":1}
I20260806 09:08:43.708288 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:43.707208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:54787)
0806 09:08:43.707889 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:43.707996 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":303,"threads_started":1}
I20260806 09:08:44.207322 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:44.206274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:39237)
0806 09:08:44.206955 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:44.207074 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":307,"threads_started":1}
I20260806 09:08:44.218138 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:44.217079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:36507)
0806 09:08:44.217784 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:44.217899 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":285,"threads_started":1}
I20260806 09:08:44.710357 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:44.709429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:43893)
0806 09:08:44.709985 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:44.710104 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260806 09:08:45.209360 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:45.208450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:47281)
0806 09:08:45.209000 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:45.209100 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260806 09:08:45.220427 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:45.219430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:48839)
0806 09:08:45.220040 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:45.220198 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":359,"threads_started":1}
I20260806 09:08:45.712442 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:45.711516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:53635)
0806 09:08:45.712065 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:45.712174 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260806 09:08:46.211613 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:46.210614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:47193)
0806 09:08:46.211310 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:46.211420 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":347,"threads_started":1}
I20260806 09:08:46.222484 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:46.221594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:59589)
0806 09:08:46.222206 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:46.222315 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260806 09:08:46.229087 13417 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:46.229415 13417 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:46.230230 13417 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
W20260806 09:08:46.231308 12997 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
W20260806 09:08:46.231498 12998 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:46.231662 12998 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70018f3b149240398798fdcddd320b81; no voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356
I20260806 09:08:46.231928 13417 raft_consensus.cc:2726] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:08:46.714417 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:46.713623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:51309)
0806 09:08:46.714126 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:46.714219 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260806 09:08:47.213887 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:47.212963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:46427)
0806 09:08:47.213477 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:47.213590 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":255,"threads_started":1}
I20260806 09:08:47.224578 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:47.223676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:34637)
0806 09:08:47.224278 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:47.224383 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260806 09:08:47.716661 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:47.715710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:35563)
0806 09:08:47.716358 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:47.716466 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":293,"threads_started":1}
I20260806 09:08:48.215886 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:48.215014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:39527)
0806 09:08:48.215622 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:48.215716 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
I20260806 09:08:48.226610 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:48.225641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:56887)
0806 09:08:48.226305 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:48.226395 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
I20260806 09:08:48.718781 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:48.717797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:54723)
0806 09:08:48.718438 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:48.718522 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":390,"threads_started":1}
I20260806 09:08:49.217950 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:49.216981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:49925)
0806 09:08:49.217628 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:49.217731 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":381,"threads_started":1}
I20260806 09:08:49.228669 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:49.227768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:51537)
0806 09:08:49.228281 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:49.228455 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":257,"threads_started":1}
I20260806 09:08:49.407763 13427 raft_consensus.cc:482] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:49.408037 13427 raft_consensus.cc:504] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70018f3b149240398798fdcddd320b81" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43255 } } peers { permanent_uuid: "bc75f05864744604a399396a8f14e356" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38385 } } peers { permanent_uuid: "83fc57e5ef6041dcb11dd6fe5aedc2cc" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 45025 } }
I20260806 09:08:49.408807 13427 leader_election.cc:288] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc75f05864744604a399396a8f14e356 (127.27.187.130:38385), 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025)
W20260806 09:08:49.409955 12997 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc75f05864744604a399396a8f14e356 (127.27.187.130:38385): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
W20260806 09:08:49.410288 12998 leader_election.cc:341] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83fc57e5ef6041dcb11dd6fe5aedc2cc (127.27.187.131:45025): Not found: Tablet not found: 8a0376f858254005915c2e2728de4cc3
I20260806 09:08:49.410449 12998 leader_election.cc:302] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70018f3b149240398798fdcddd320b81; no voters: 83fc57e5ef6041dcb11dd6fe5aedc2cc, bc75f05864744604a399396a8f14e356
I20260806 09:08:49.410802 13427 raft_consensus.cc:2726] T 8a0376f858254005915c2e2728de4cc3 P 70018f3b149240398798fdcddd320b81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:08:49.720690 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:49.719813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:53207)
0806 09:08:49.720344 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:49.720458 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":355,"threads_started":1}
I20260806 09:08:50.220264 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:50.219331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:50845)
0806 09:08:50.219937 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:50.220034 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":369,"threads_started":1}
I20260806 09:08:50.230942 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:50.229962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:59237)
0806 09:08:50.230593 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:50.230757 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":348,"threads_started":1}
I20260806 09:08:50.722910 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:50.721889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:58409)
0806 09:08:50.722501 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:50.722600 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":307,"threads_started":1}
I20260806 09:08:51.222883 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:51.221885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:38781)
0806 09:08:51.222464 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:51.222612 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":288,"threads_started":1}
I20260806 09:08:51.232863 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:51.231992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:56865)
0806 09:08:51.232559 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:51.232657 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
I20260806 09:08:51.725658 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:51.724275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.130:43559)
0806 09:08:51.724990 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:51.725323 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
I20260806 09:08:52.209317 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 12938
I20260806 09:08:52.225337 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:52.223961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.129:45807)
0806 09:08:52.224773 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:52.224970 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":474,"threads_started":1}
I20260806 09:08:52.231937 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:52.210565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.27.187.129:43255)
0806 09:08:52.211225 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:52.211233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:52.211265 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:08:52.211338 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:52.211345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:52.211570 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:52.211783 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:52.213596 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.214625 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.216110 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.216722 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.216886 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:52.224388 (+ 7502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:52.224412 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:52.224429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:52.224472 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:52.228409 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.229388 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.229397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.229399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.229563 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.230019 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.230025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.230028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.230444 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:52.230806 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:52.231154 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 09:08:52.231566 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":328,"threads_started":1}
I20260806 09:08:52.233669 13106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:52.235551 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:52.234180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40813 (local address 127.27.187.131:42651)
0806 09:08:52.234941 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:52.235261 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.190:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":313,"threads_started":1}
I20260806 09:08:52.331040 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 12938
I20260806 09:08:52.352600 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13115
I20260806 09:08:52.369474 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:52.353602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.27.187.130:38385)
0806 09:08:52.354289 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:52.354293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:52.354310 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:08:52.354383 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:52.354387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:52.354458 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:52.354561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:52.356295 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.357285 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.358629 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.359534 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.359721 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:52.362885 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:52.362892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:52.362896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:52.362939 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:52.366000 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.367291 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.367296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.367298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.367495 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.367926 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.367929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.367932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.368265 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:52.368506 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:52.368819 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:08:52.369016 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":397,"threads_started":1}
I20260806 09:08:52.370324 13240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:52.462143 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13115
I20260806 09:08:52.484218 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13248
I20260806 09:08:52.501304 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:52.485156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.27.187.131:45025)
0806 09:08:52.485852 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:52.485857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:52.485876 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:08:52.485936 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:52.485940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:52.486016 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:52.486115 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:52.487812 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.488780 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.490217 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:52.490890 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:52.491008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:52.493967 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:52.493973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:52.493977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:52.494020 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:52.497794 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.498923 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.498929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.498931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.499140 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:52.499769 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:52.499774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:52.499776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:52.500108 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:52.500373 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:52.500761 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 09:08:52.500964 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":405,"threads_started":1}
I20260806 09:08:52.502143 13373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:52.592829 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13248
2026-08-06T09:08:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65624 ms)
[----------] 30 tests from SecurityITest (329035 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:08:52 dist-test-slave-8tdl krb5kdc[13445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:08:52 dist-test-slave-8tdl krb5kdc[13445](info): set up 2 sockets
Aug 06 09:08:52 dist-test-slave-8tdl krb5kdc[13445](info): commencing operation
krb5kdc: starting...
W20260806 09:08:53.801887 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:08:54 dist-test-slave-8tdl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:08:54.143046 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40483
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:40483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:08:54.515347 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:54.515684 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:54.515857 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:54.523653 13458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:08:54.523801 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:54.523906 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:54.524039 13458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:08:54.524236 13458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:08:54.535014 13458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:40483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:40483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:54.536702 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:54.538437 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:54.551990 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:54.551884 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:54.551947 13466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:54.553493 13458 server_base.cc:1034] running on GCE node
I20260806 09:08:54.554502 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:54.556483 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:54.557785 13458 hybrid_clock.cc:648] HybridClock initialized: now 1786007334557738 us; error 40 us; skew 500 ppm
I20260806 09:08:54.558244 13458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:54 dist-test-slave-8tdl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:54.561952 13458 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:08:54.564064 13458 webserver.cc:460] Webserver started at http://127.27.187.190:39503/ using document root <none> and password file <none>
I20260806 09:08:54.564857 13458 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:54.565030 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:54.565377 13458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:54.568789 13458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "025ffe73bc564147a4ea624d6af526c8"
format_stamp: "Formatted at 2026-08-06 09:08:54 on dist-test-slave-8tdl"
server_key: "ba41e9995fa22ef3f74f4bd2eaaef98c"
server_key_iv: "c49c0a54e7843e35bd1ac2768f5fdcb5"
server_key_version: "encryptionkey@0"
I20260806 09:08:54.569694 13458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "025ffe73bc564147a4ea624d6af526c8"
format_stamp: "Formatted at 2026-08-06 09:08:54 on dist-test-slave-8tdl"
server_key: "ba41e9995fa22ef3f74f4bd2eaaef98c"
server_key_iv: "c49c0a54e7843e35bd1ac2768f5fdcb5"
server_key_version: "encryptionkey@0"
I20260806 09:08:54.575734 13458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:08:54.579876 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:54.581607 13458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:54.581866 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "025ffe73bc564147a4ea624d6af526c8"
format_stamp: "Formatted at 2026-08-06 09:08:54 on dist-test-slave-8tdl"
server_key: "ba41e9995fa22ef3f74f4bd2eaaef98c"
server_key_iv: "c49c0a54e7843e35bd1ac2768f5fdcb5"
server_key_version: "encryptionkey@0"
I20260806 09:08:54.582131 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:54.611757 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:54.617394 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:54.617739 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:54.639696 13458 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:40483
I20260806 09:08:54.639730 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:40483 every 8 connection(s)
I20260806 09:08:54.641274 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:08:54.645860 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:54.649133 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13458
I20260806 09:08:54.649396 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:08:54.650287 28398 external_mini_cluster.cc:1428] Setting key 906bc3b3758804d9dd6561f8c084d3a6
I20260806 09:08:54.659503 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: Bootstrap starting.
I20260806 09:08:54.662940 13526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:54.664295 13526 log.cc:824] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:08:54 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:54.668036 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: No bootstrap required, opened a new log
I20260806 09:08:54.677165 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:54.652873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.27.187.190:40483)
0806 09:08:54.653848 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:54.653863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:54.653909 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:08:54.655879 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:54.655906 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:54.656390 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:54.656877 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:54.658528 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:54.660113 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:54.661558 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:54.662269 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:54.662478 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:54.667893 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:54.667903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:54.667919 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:54.667962 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:54.671973 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:54.673002 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:54.673011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:54.673019 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:54.673248 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:54.673741 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:54.673748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:54.673757 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:54.674471 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:54.674855 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:54.675416 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 09:08:54.675954 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":443,"threads_started":1}
I20260806 09:08:54.677198 13526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } }
I20260806 09:08:54.677690 13526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:54.677819 13526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025ffe73bc564147a4ea624d6af526c8, State: Initialized, Role: FOLLOWER
I20260806 09:08:54.678535 13526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } }
I20260806 09:08:54.678867 13526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:08:54.679031 13526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:08:54.679239 13526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:54.681923 13526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } }
I20260806 09:08:54.682487 13526 leader_election.cc:302] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025ffe73bc564147a4ea624d6af526c8; no voters:
I20260806 09:08:54.683223 13526 leader_election.cc:288] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:08:54.683390 13531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:08:54.684203 13531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [term 1 LEADER]: Becoming Leader. State: Replica: 025ffe73bc564147a4ea624d6af526c8, State: Running, Role: LEADER
I20260806 09:08:54.684782 13531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } }
I20260806 09:08:54.685734 13526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:08:54.688411 13532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025ffe73bc564147a4ea624d6af526c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } } }
I20260806 09:08:54.688438 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025ffe73bc564147a4ea624d6af526c8. Latest consensus state: current_term: 1 leader_uuid: "025ffe73bc564147a4ea624d6af526c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025ffe73bc564147a4ea624d6af526c8" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 40483 } } }
I20260806 09:08:54.688862 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:54.688867 13532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:54.693475 13538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:08:54.699738 13538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:08:54.711984 13538 catalog_manager.cc:1347] Generated new cluster ID: 10998b6205464fb8823739e356d9811f
I20260806 09:08:54.712188 13538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:08:54.728533 13538 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:08:54.730199 13538 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:08:54.749578 13538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: Generated new TSK 0
I20260806 09:08:54.750654 13538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:08:54.827386 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:08:55.212977 13552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:55.213303 13552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:55.213471 13552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:55.221212 13552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:55.221342 13552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:55.221555 13552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:08:55.232182 13552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:55.233690 13552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:55.235363 13552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:55.250752 13557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:55.250759 13558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:55.250794 13560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:55.252045 13552 server_base.cc:1034] running on GCE node
I20260806 09:08:55.252821 13552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:55.253935 13552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:55.255244 13552 hybrid_clock.cc:648] HybridClock initialized: now 1786007335255163 us; error 40 us; skew 500 ppm
I20260806 09:08:55.255611 13552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:55 dist-test-slave-8tdl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:55.259244 13552 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:08:55.261003 13552 webserver.cc:460] Webserver started at http://127.27.187.129:37597/ using document root <none> and password file <none>
I20260806 09:08:55.261823 13552 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:55.262012 13552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:55.262367 13552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:55.265668 13552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "ec39fe528ffa4c7ebcf25ade272d69fc"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "eaebed5980c056978c974ac80efdc82a"
server_key_iv: "aa5a4c4baba41165758ffb69628289dc"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.266588 13552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "ec39fe528ffa4c7ebcf25ade272d69fc"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "eaebed5980c056978c974ac80efdc82a"
server_key_iv: "aa5a4c4baba41165758ffb69628289dc"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.273326 13552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:08:55.278033 13567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.279847 13552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:08:55.280087 13552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "ec39fe528ffa4c7ebcf25ade272d69fc"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "eaebed5980c056978c974ac80efdc82a"
server_key_iv: "aa5a4c4baba41165758ffb69628289dc"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.280356 13552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:55.296051 13552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:55.301374 13552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:55.301743 13552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:55.303102 13552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:55.304756 13552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:55.304878 13552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.305058 13552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:55.305140 13552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.344156 13552 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:46439
I20260806 09:08:55.344189 13680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:46439 every 8 connection(s)
I20260806 09:08:55.345706 13552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:08:55.354421 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13552
I20260806 09:08:55.354713 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:08:55.355409 28398 external_mini_cluster.cc:1428] Setting key c0c1c773aaea7cbda6bd60e224d7e200
Aug 06 09:08:55 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:55.368980 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:55.348883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:37991 (local address 127.27.187.190:40483)
0806 09:08:55.349365 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:55.349370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:55.350849 (+ 1479us) server_negotiation.cc:407] Connection header received
0806 09:08:55.352873 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:55.352879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:55.352981 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:55.353106 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:55.354989 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.355803 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.357847 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.358266 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.358415 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:55.362326 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:55.362332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:55.362335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:55.362373 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:55.365434 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:55.366346 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:55.366351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:55.366353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:55.366484 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:55.367025 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:55.367032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:55.367040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:55.367352 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:55.367523 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:55.368329 (+ 806us) server_negotiation.cc:299] Negotiation successful
0806 09:08:55.368596 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
I20260806 09:08:55.370031 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:55.349720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40483 (local address 127.27.187.129:37991)
0806 09:08:55.350582 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:55.350623 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:55.352547 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:55.353381 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:55.353391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:55.354012 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:55.354754 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:55.354796 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.355993 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.355999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:55.357657 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:55.357679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.358525 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.358533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:55.358890 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:55.359654 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:55.359681 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:55.362042 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:55.365642 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:55.365653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:55.365680 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:55.366189 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:55.366615 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:55.366620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:55.366624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:55.366894 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:55.367491 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:55.367499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:55.367898 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 09:08:55.368213 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 09:08:55.368824 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":370,"threads_started":1}
I20260806 09:08:55.371831 13681 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40483
I20260806 09:08:55.372236 13681 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:55.373180 13681 heartbeater.cc:511] Master 127.27.187.190:40483 requested a full tablet report, sending...
I20260806 09:08:55.375680 13491 ts_manager.cc:194] Registered new tserver with Master: ec39fe528ffa4c7ebcf25ade272d69fc (127.27.187.129:46439)
I20260806 09:08:55.377223 13491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:37991
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:08:55.431947 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:08:55.816573 13687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:55.816895 13687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:55.817078 13687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:55.824882 13687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:55.825026 13687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:55.825235 13687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:08:55.835675 13687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:55.837137 13687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:55.838644 13687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:55.854395 13695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:55.854668 13692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:55.854636 13693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:55.855202 13687 server_base.cc:1034] running on GCE node
I20260806 09:08:55.856082 13687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:55.857199 13687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:55.858438 13687 hybrid_clock.cc:648] HybridClock initialized: now 1786007335858367 us; error 62 us; skew 500 ppm
I20260806 09:08:55.858898 13687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:55 dist-test-slave-8tdl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:55.862625 13687 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:08:55.864589 13687 webserver.cc:460] Webserver started at http://127.27.187.130:45849/ using document root <none> and password file <none>
I20260806 09:08:55.865393 13687 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:55.865561 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:55.865921 13687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:55.869308 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "e712139886b84b6390b4c5dd1ff10d45"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "b251586f562a05a1f862d589e2e6c78b"
server_key_iv: "c904da39c2408ab14efaa950e35c29f1"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.870240 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "e712139886b84b6390b4c5dd1ff10d45"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "b251586f562a05a1f862d589e2e6c78b"
server_key_iv: "c904da39c2408ab14efaa950e35c29f1"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.876221 13687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:08:55.880438 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.882229 13687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:55.882459 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "e712139886b84b6390b4c5dd1ff10d45"
format_stamp: "Formatted at 2026-08-06 09:08:55 on dist-test-slave-8tdl"
server_key: "b251586f562a05a1f862d589e2e6c78b"
server_key_iv: "c904da39c2408ab14efaa950e35c29f1"
server_key_version: "encryptionkey@0"
I20260806 09:08:55.882748 13687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:55.915572 13687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:55.920833 13687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:55.921175 13687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:55.922325 13687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:55.923884 13687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:55.924001 13687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.924171 13687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:55.924260 13687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:55.961485 13687 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:42219
I20260806 09:08:55.961584 13818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:42219 every 8 connection(s)
I20260806 09:08:55.963012 13687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:08:55.968592 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13687
I20260806 09:08:55.968849 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:08:55.969369 28398 external_mini_cluster.cc:1428] Setting key 987b72457c002f8bd248ffa3c8cceda1
Aug 06 09:08:55 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:55.987689 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:55.965956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:42185 (local address 127.27.187.190:40483)
0806 09:08:55.966469 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:55.966474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:55.967714 (+ 1240us) server_negotiation.cc:407] Connection header received
0806 09:08:55.969899 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:55.969905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:55.970011 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:55.970150 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:55.972165 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.972950 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.974669 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.975204 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.975351 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:55.979894 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:55.979901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:55.979905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:55.979953 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:55.983407 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:55.984634 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:55.984640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:55.984643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:55.984846 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:55.985496 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:55.985501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:55.985505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:55.985872 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:55.986056 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:55.986970 (+ 914us) server_negotiation.cc:299] Negotiation successful
0806 09:08:55.987270 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":254,"threads_started":1}
I20260806 09:08:55.989054 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:55.966610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40483 (local address 127.27.187.130:42185)
0806 09:08:55.967520 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:55.967567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:55.969570 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:55.970465 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:55.970478 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:55.971341 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:55.971963 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:55.971983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.973146 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.973150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:55.974485 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:55.974499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:55.975456 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:55.975463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:55.975729 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:55.976544 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:55.976569 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:55.979540 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:55.983608 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:55.983621 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:55.983649 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:55.984165 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:55.985009 (+ 844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:55.985014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:55.985020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:55.985348 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:55.986010 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:55.986019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:55.986507 (+ 488us) client_negotiation.cc:769] Sending connection context
0806 09:08:55.986883 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 09:08:55.987577 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":400,"threads_started":1}
I20260806 09:08:55.991411 13819 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40483
I20260806 09:08:55.991803 13819 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:55.992848 13819 heartbeater.cc:511] Master 127.27.187.190:40483 requested a full tablet report, sending...
I20260806 09:08:55.994922 13491 ts_manager.cc:194] Registered new tserver with Master: e712139886b84b6390b4c5dd1ff10d45 (127.27.187.130:42219)
I20260806 09:08:55.996258 13491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:42185
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:08:56.055912 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:08:56.381453 13681 heartbeater.cc:503] Master 127.27.187.190:40483 was elected leader, sending a full tablet report...
W20260806 09:08:56.427595 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:56.427978 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:56.428153 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:56.435952 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:56.436129 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:56.436326 13825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:08:56.446587 13825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:40483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:56.448117 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:56.449545 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:56.464951 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:56.464962 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:56.465195 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:56.465875 13825 server_base.cc:1034] running on GCE node
I20260806 09:08:56.466629 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:56.467885 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:56.469185 13825 hybrid_clock.cc:648] HybridClock initialized: now 1786007336469092 us; error 56 us; skew 500 ppm
I20260806 09:08:56.469558 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:08:56 dist-test-slave-8tdl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:08:56.473389 13825 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:08:56.475260 13825 webserver.cc:460] Webserver started at http://127.27.187.131:35769/ using document root <none> and password file <none>
I20260806 09:08:56.476065 13825 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:56.476217 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:56.476598 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:56.480270 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "ed02ed0502d442c787f9e658fd62ef4c"
format_stamp: "Formatted at 2026-08-06 09:08:56 on dist-test-slave-8tdl"
server_key: "b193e7ea35f8111b1824bdc0a615ccfb"
server_key_iv: "560665cc0cde4061b7420b136b1e708b"
server_key_version: "encryptionkey@0"
I20260806 09:08:56.481227 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "ed02ed0502d442c787f9e658fd62ef4c"
format_stamp: "Formatted at 2026-08-06 09:08:56 on dist-test-slave-8tdl"
server_key: "b193e7ea35f8111b1824bdc0a615ccfb"
server_key_iv: "560665cc0cde4061b7420b136b1e708b"
server_key_version: "encryptionkey@0"
I20260806 09:08:56.488471 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260806 09:08:56.492745 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:56.494778 13825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:08:56.494987 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "ed02ed0502d442c787f9e658fd62ef4c"
format_stamp: "Formatted at 2026-08-06 09:08:56 on dist-test-slave-8tdl"
server_key: "b193e7ea35f8111b1824bdc0a615ccfb"
server_key_iv: "560665cc0cde4061b7420b136b1e708b"
server_key_version: "encryptionkey@0"
I20260806 09:08:56.495234 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:56.529018 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:56.533974 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:56.534308 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:56.535563 13825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:56.537071 13825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:56.537153 13825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:56.537313 13825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:56.537367 13825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:56.575580 13825 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:44919
I20260806 09:08:56.575620 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:44919 every 8 connection(s)
I20260806 09:08:56.577105 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:08:56.582044 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 13825
I20260806 09:08:56.582324 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:08:56.583158 28398 external_mini_cluster.cc:1428] Setting key 9bb9cdc01fd23b31320e97ea8c3fe6d1
Aug 06 09:08:56 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:08:56.602908 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.580194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:35081 (local address 127.27.187.190:40483)
0806 09:08:56.580690 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.580697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.582196 (+ 1499us) server_negotiation.cc:407] Connection header received
0806 09:08:56.585040 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.585044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.585135 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.585250 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:56.587561 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.588644 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.590430 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.591086 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.591233 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.595444 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:56.595451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:56.595454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:56.595494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:56.599034 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.600104 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.600110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.600113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.600300 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.600872 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.600876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.600878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.601203 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:56.601341 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.602272 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.602477 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":223,"threads_started":1}
I20260806 09:08:56.603883 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.580890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:40483 (local address 127.27.187.131:35081)
0806 09:08:56.581934 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.581999 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.584703 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.585608 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.585622 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.586344 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:56.587375 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.587398 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.588844 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.588849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.590241 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.590256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.591344 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.591350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.591585 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.592508 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:08:56.592535 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:56.595175 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:56.599216 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:56.599224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:56.599241 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:56.599931 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:56.600459 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:08:56.600464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:08:56.600467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:08:56.600740 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:08:56.601350 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:56.601356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:56.601761 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.602040 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.602534 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":459,"threads_started":1}
I20260806 09:08:56.605398 13956 heartbeater.cc:347] Connected to a master server at 127.27.187.190:40483
I20260806 09:08:56.605799 13956 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:56.606642 13956 heartbeater.cc:511] Master 127.27.187.190:40483 requested a full tablet report, sending...
I20260806 09:08:56.608495 13491 ts_manager.cc:194] Registered new tserver with Master: ed02ed0502d442c787f9e658fd62ef4c (127.27.187.131:44919)
I20260806 09:08:56.609510 13491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:35081
I20260806 09:08:56.612759 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:08:56.634646 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.619400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.27.187.190:40483)
0806 09:08:56.619715 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.619723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.620029 (+ 306us) server_negotiation.cc:407] Connection header received
0806 09:08:56.620265 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.620271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.620373 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.620491 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:56.622153 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.623127 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.624812 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.625286 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.625434 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.628329 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:56.628335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:56.628338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:56.628376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:56.631565 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.632392 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.632398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.632403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.632549 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.632955 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.632963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.632967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.633354 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:56.633629 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.634003 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.634286 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:08:56.639647 13491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:08:56.647886 13491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:08:56.674017 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.663659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.27.187.130:42219)
0806 09:08:56.664565 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.664575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.664607 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:08:56.664994 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.665003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.665295 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.666282 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:56.666550 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.668307 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.670803 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.672504 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.672718 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.672930 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.673486 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.673684 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":540,"threads_started":1}
I20260806 09:08:56.674033 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.663465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:42219 (local address 127.0.0.1:49094)
0806 09:08:56.664383 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.664427 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.664629 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.665631 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.665642 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.665706 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:56.666143 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.666160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.668491 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.668498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.670568 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.670584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.672742 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.672748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.673003 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.673053 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.673241 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.673512 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":27,"thread_start_us":386,"threads_started":1}
I20260806 09:08:56.674472 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.665342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.27.187.131:44919)
0806 09:08:56.666195 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.666203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.666249 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:08:56.666365 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.666376 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.666669 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.666988 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:56.667777 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.669396 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.671580 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.673156 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.673460 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.673653 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.673911 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.674094 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":513,"threads_started":1}
I20260806 09:08:56.674579 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.664789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:44919 (local address 127.0.0.1:60374)
0806 09:08:56.665399 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.665420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.665587 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.666961 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.666966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.666994 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:56.667481 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.667495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.669548 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.669554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.671372 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.671387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.673473 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.673479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.673705 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.673742 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.673858 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.674016 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":297,"threads_started":1}
I20260806 09:08:56.679142 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.666231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46439 (local address 127.0.0.1:34462)
0806 09:08:56.667881 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.667902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.668027 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.668978 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.668983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.669010 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:56.671050 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.671064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.673227 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.673234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.676180 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.676197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.678222 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.678230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.678407 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.678440 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.678542 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.678723 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"mutex_wait_us":1,"thread_start_us":1367,"threads_started":1}
I20260806 09:08:56.679222 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.665888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.27.187.129:46439)
0806 09:08:56.666847 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.666856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.668172 (+ 1316us) server_negotiation.cc:407] Connection header received
0806 09:08:56.668299 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.668308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.668623 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.668993 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:56.671274 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.673046 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.676380 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.677956 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.678210 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.678387 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.678639 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.678844 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":546,"threads_started":1}
I20260806 09:08:56.679185 13889 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3492d31ae04499b73589f818feced3 (DEFAULT_TABLE table=test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:56.679216 13751 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3492d31ae04499b73589f818feced3 (DEFAULT_TABLE table=test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:56.680933 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3492d31ae04499b73589f818feced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:56.681679 13751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3492d31ae04499b73589f818feced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:56.683183 13616 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3492d31ae04499b73589f818feced3 (DEFAULT_TABLE table=test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:56.684514 13616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3492d31ae04499b73589f818feced3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:56.697728 13976 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Bootstrap starting.
I20260806 09:08:56.700093 13977 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Bootstrap starting.
I20260806 09:08:56.700986 13978 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Bootstrap starting.
I20260806 09:08:56.703444 13976 tablet_bootstrap.cc:654] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:56.704799 13976 log.cc:824] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:56.704918 13978 tablet_bootstrap.cc:654] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:56.705060 13977 tablet_bootstrap.cc:654] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:56.706400 13978 log.cc:824] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:56.706400 13977 log.cc:824] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:56.709062 13976 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: No bootstrap required, opened a new log
I20260806 09:08:56.709520 13976 ts_tablet_manager.cc:1397] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260806 09:08:56.709746 13978 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: No bootstrap required, opened a new log
I20260806 09:08:56.709975 13977 tablet_bootstrap.cc:492] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: No bootstrap required, opened a new log
I20260806 09:08:56.710098 13978 ts_tablet_manager.cc:1397] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:08:56.710312 13977 ts_tablet_manager.cc:1397] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 09:08:56.718168 13978 raft_consensus.cc:348] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.718628 13978 raft_consensus.cc:374] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:56.718784 13978 raft_consensus.cc:729] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed02ed0502d442c787f9e658fd62ef4c, State: Initialized, Role: FOLLOWER
I20260806 09:08:56.719519 13978 consensus_queue.cc:260] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.721920 13977 raft_consensus.cc:348] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.721920 13976 raft_consensus.cc:348] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.722448 13976 raft_consensus.cc:374] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:56.722492 13977 raft_consensus.cc:374] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:56.722645 13976 raft_consensus.cc:729] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec39fe528ffa4c7ebcf25ade272d69fc, State: Initialized, Role: FOLLOWER
I20260806 09:08:56.722638 13977 raft_consensus.cc:729] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e712139886b84b6390b4c5dd1ff10d45, State: Initialized, Role: FOLLOWER
I20260806 09:08:56.723091 13956 heartbeater.cc:503] Master 127.27.187.190:40483 was elected leader, sending a full tablet report...
I20260806 09:08:56.723410 13977 consensus_queue.cc:260] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.723413 13976 consensus_queue.cc:260] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.723850 13978 ts_tablet_manager.cc:1428] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260806 09:08:56.725975 13976 ts_tablet_manager.cc:1428] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260806 09:08:56.726161 13977 ts_tablet_manager.cc:1428] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260806 09:08:56.726508 13819 heartbeater.cc:503] Master 127.27.187.190:40483 was elected leader, sending a full tablet report...
I20260806 09:08:56.797572 13982 raft_consensus.cc:482] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:08:56.797883 13982 raft_consensus.cc:504] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.799916 13982 leader_election.cc:288] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e712139886b84b6390b4c5dd1ff10d45 (127.27.187.130:42219), ec39fe528ffa4c7ebcf25ade272d69fc (127.27.187.129:46439)
I20260806 09:08:56.805902 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.800439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:56071 (local address 127.27.187.130:42219)
0806 09:08:56.800690 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.800697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.800722 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:08:56.800850 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.800856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.800968 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.801101 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:56.801579 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.802327 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.804092 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.804933 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.805110 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.805189 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.805495 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.805627 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:08:56.805902 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.800176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:42219 (local address 127.27.187.131:56071)
0806 09:08:56.800424 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.800444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.800689 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.801152 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.801157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.801192 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:56.801413 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.801422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.802495 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.802501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.803901 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.803915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.805101 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.805107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.805316 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.805349 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.805461 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.805599 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:08:56.807137 13771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3492d31ae04499b73589f818feced3" candidate_uuid: "ed02ed0502d442c787f9e658fd62ef4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e712139886b84b6390b4c5dd1ff10d45" is_pre_election: true
I20260806 09:08:56.807657 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.800886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:41645 (local address 127.27.187.129:46439)
0806 09:08:56.801465 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.801473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.801646 (+ 173us) server_negotiation.cc:407] Connection header received
0806 09:08:56.801881 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.801892 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.801957 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.802076 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:56.802763 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.803568 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.805416 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.806578 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.806814 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.806899 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.807215 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.807382 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404}
I20260806 09:08:56.807730 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.800743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:46439 (local address 127.27.187.131:41645)
0806 09:08:56.801517 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:56.801547 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:56.801711 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:56.802091 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:56.802096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:56.802121 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:56.802565 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.802580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.803740 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.803746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.805224 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:56.805236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.806832 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.806839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:56.807027 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.807056 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:08:56.807185 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:08:56.807340 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":373,"threads_started":1}
I20260806 09:08:56.807840 13771 raft_consensus.cc:2454] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed02ed0502d442c787f9e658fd62ef4c in term 0.
I20260806 09:08:56.808652 13841 leader_election.cc:302] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e712139886b84b6390b4c5dd1ff10d45, ed02ed0502d442c787f9e658fd62ef4c; no voters:
I20260806 09:08:56.809113 13982 raft_consensus.cc:2781] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:08:56.809258 13982 raft_consensus.cc:482] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:08:56.809368 13982 raft_consensus.cc:3037] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:56.809159 13636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3492d31ae04499b73589f818feced3" candidate_uuid: "ed02ed0502d442c787f9e658fd62ef4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" is_pre_election: true
I20260806 09:08:56.809722 13636 raft_consensus.cc:2454] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed02ed0502d442c787f9e658fd62ef4c in term 0.
I20260806 09:08:56.812325 13982 raft_consensus.cc:504] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.813153 13982 leader_election.cc:288] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [CANDIDATE]: Term 1 election: Requested vote from peers e712139886b84b6390b4c5dd1ff10d45 (127.27.187.130:42219), ec39fe528ffa4c7ebcf25ade272d69fc (127.27.187.129:46439)
I20260806 09:08:56.813760 13771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3492d31ae04499b73589f818feced3" candidate_uuid: "ed02ed0502d442c787f9e658fd62ef4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e712139886b84b6390b4c5dd1ff10d45"
I20260806 09:08:56.813786 13636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3492d31ae04499b73589f818feced3" candidate_uuid: "ed02ed0502d442c787f9e658fd62ef4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc"
I20260806 09:08:56.814007 13771 raft_consensus.cc:3037] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:56.814026 13636 raft_consensus.cc:3037] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:56.816721 13771 raft_consensus.cc:2454] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed02ed0502d442c787f9e658fd62ef4c in term 1.
I20260806 09:08:56.817240 13636 raft_consensus.cc:2454] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed02ed0502d442c787f9e658fd62ef4c in term 1.
I20260806 09:08:56.817271 13841 leader_election.cc:302] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e712139886b84b6390b4c5dd1ff10d45, ed02ed0502d442c787f9e658fd62ef4c; no voters:
I20260806 09:08:56.817629 13982 raft_consensus.cc:2781] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:08:56.818282 13982 raft_consensus.cc:686] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 1 LEADER]: Becoming Leader. State: Replica: ed02ed0502d442c787f9e658fd62ef4c, State: Running, Role: LEADER
I20260806 09:08:56.818857 13982 consensus_queue.cc:237] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } }
I20260806 09:08:56.824427 13490 catalog_manager.cc:5495] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c reported cstate change: term changed from 0 to 1, leader changed from <none> to ed02ed0502d442c787f9e658fd62ef4c (127.27.187.131). New cstate: current_term: 1 leader_uuid: "ed02ed0502d442c787f9e658fd62ef4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed02ed0502d442c787f9e658fd62ef4c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 } health_report { overall_health: UNKNOWN } } }
W20260806 09:08:56.829012 13957 tablet.cc:2369] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:08:56.848865 13682 tablet.cc:2369] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:08:56.879444 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.873528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.27.187.131:44919)
0806 09:08:56.873721 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.873725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.873744 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:08:56.873925 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.873929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.874012 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.874261 (+ 249us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:08:56.874734 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.875461 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.877102 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.877538 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.877689 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.878207 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:08:56.878848 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:08:56.879021 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.879074 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.879160 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:08:56.889259 13636 raft_consensus.cc:1261] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 1 FOLLOWER]: Refusing update from remote peer ed02ed0502d442c787f9e658fd62ef4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:08:56.889276 13771 raft_consensus.cc:1261] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 1 FOLLOWER]: Refusing update from remote peer ed02ed0502d442c787f9e658fd62ef4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:08:56.890347 13986 consensus_queue.cc:1035] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e712139886b84b6390b4c5dd1ff10d45" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:08:56.890887 13982 consensus_queue.cc:1035] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec39fe528ffa4c7ebcf25ade272d69fc" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:08:56.901059 13989 mvcc.cc:205] Tried to move back new op lower bound from 7315486051886338048 to 7315486051614441472. Current Snapshot: MvccSnapshot[applied={T|T < 7315486051886338048}]
I20260806 09:08:56.925122 13490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:08:56.925567 13490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:08:56.932252 13490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 025ffe73bc564147a4ea624d6af526c8: Sending DeleteTablet for 3 replicas of tablet 0b3492d31ae04499b73589f818feced3
I20260806 09:08:56.933935 13751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3492d31ae04499b73589f818feced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:56 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:49094
I20260806 09:08:56.934126 13616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3492d31ae04499b73589f818feced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:56 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:34462
I20260806 09:08:56.934273 13614 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3492d31ae04499b73589f818feced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:56 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:34462
I20260806 09:08:56.935322 13889 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3492d31ae04499b73589f818feced3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:08:56 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:60374
I20260806 09:08:56.936743 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13552
I20260806 09:08:56.964671 14001 tablet_replica.cc:330] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: stopping tablet replica
I20260806 09:08:56.968910 14001 raft_consensus.cc:2229] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:08:56.969831 13999 tablet_replica.cc:330] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: stopping tablet replica
I20260806 09:08:56.970310 14001 raft_consensus.cc:2258] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:08:56.970376 13999 raft_consensus.cc:2229] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:08:56.971808 14000 tablet_replica.cc:330] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: stopping tablet replica
I20260806 09:08:56.972276 14000 raft_consensus.cc:2229] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:08:56.972903 14000 raft_consensus.cc:2258] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:08:56.973196 14001 ts_tablet_manager.cc:1905] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:08:56.973726 13999 raft_consensus.cc:2258] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:08:56.975688 14000 ts_tablet_manager.cc:1905] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:08:56 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:08:56.981993 14001 ts_tablet_manager.cc:1918] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:08:56.982208 14001 log.cc:1196] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/0b3492d31ae04499b73589f818feced3
I20260806 09:08:56.982806 14001 ts_tablet_manager.cc:1939] T 0b3492d31ae04499b73589f818feced3 P ed02ed0502d442c787f9e658fd62ef4c: Deleting consensus metadata
I20260806 09:08:56.983976 14000 ts_tablet_manager.cc:1918] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:08:56.984171 14000 log.cc:1196] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/0b3492d31ae04499b73589f818feced3
I20260806 09:08:56.984804 14000 ts_tablet_manager.cc:1939] T 0b3492d31ae04499b73589f818feced3 P ec39fe528ffa4c7ebcf25ade272d69fc: Deleting consensus metadata
I20260806 09:08:56.987469 13999 ts_tablet_manager.cc:1905] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:08:56.991406 13475 catalog_manager.cc:4841] TS ed02ed0502d442c787f9e658fd62ef4c (127.27.187.131:44919): tablet 0b3492d31ae04499b73589f818feced3 (table test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]) successfully deleted
I20260806 09:08:56.992769 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:56.938896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.27.187.129:46439)
0806 09:08:56.939162 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:56.939172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:56.963574 (+ 24402us) server_negotiation.cc:407] Connection header received
0806 09:08:56.965436 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:56.965446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:56.965570 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:56.965681 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:56.970987 (+ 5306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.972437 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.974589 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:56.975297 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:56.975433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:56.981753 (+ 6320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:56.981775 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:56.981801 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:56.981853 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:56.987740 (+ 5887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.988977 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.988987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.988992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.989188 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:56.989691 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:56.989697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:56.989700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:56.990209 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:56.990534 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:56.992061 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0806 09:08:56.992331 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:08:56.993269 13475 catalog_manager.cc:4841] TS ec39fe528ffa4c7ebcf25ade272d69fc (127.27.187.129:46439): tablet 0b3492d31ae04499b73589f818feced3 (table test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]) successfully deleted
W20260806 09:08:56.993780 13475 catalog_manager.cc:4808] TS ec39fe528ffa4c7ebcf25ade272d69fc (127.27.187.129:46439): delete failed for tablet 0b3492d31ae04499b73589f818feced3 because the tablet was not found. No further retry: Not found: Tablet not found: 0b3492d31ae04499b73589f818feced3
I20260806 09:08:56.994982 13676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:56.995517 13999 ts_tablet_manager.cc:1918] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:08:56.995821 13999 log.cc:1196] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/0b3492d31ae04499b73589f818feced3
I20260806 09:08:56.996505 13999 ts_tablet_manager.cc:1939] T 0b3492d31ae04499b73589f818feced3 P e712139886b84b6390b4c5dd1ff10d45: Deleting consensus metadata
I20260806 09:08:56.999090 13475 catalog_manager.cc:4841] TS e712139886b84b6390b4c5dd1ff10d45 (127.27.187.130:42219): tablet 0b3492d31ae04499b73589f818feced3 (table test-table [id=05812eb3d55b4c878d3940dda0dbc8bf]) successfully deleted
I20260806 09:08:57.092067 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13552
I20260806 09:08:57.114485 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13687
Aug 06 09:08:57 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:08:57.133773 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:57.115455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.27.187.130:42219)
0806 09:08:57.115677 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:57.115684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:57.115710 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:08:57.115814 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:57.115821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:57.115919 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:57.116033 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:57.117372 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:57.118557 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:57.120215 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:57.120853 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:57.120995 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:57.125927 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:57.125947 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:57.125969 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:57.126043 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:57.130235 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:57.131265 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:57.131277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:57.131281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:57.131461 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:57.131890 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:57.131896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:57.131899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:57.132466 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:57.132789 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:57.133119 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 09:08:57.133379 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:08:57.134745 13814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:57.229421 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13687
I20260806 09:08:57.249334 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13825
Aug 06 09:08:57 dist-test-slave-8tdl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:08:57.274623 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:57.250329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.27.187.131:44919)
0806 09:08:57.250546 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:57.250553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:57.250578 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:08:57.250649 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:57.250655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:57.250777 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:57.250916 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:57.252403 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:57.253654 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:57.255186 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:57.255906 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:57.256080 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:57.266180 (+ 10100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:57.266216 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:57.266239 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:08:57.266313 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:57.270785 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:57.271740 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:57.271750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:57.271753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:57.271980 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:08:57.272418 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:08:57.272424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:08:57.272427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:08:57.272943 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:57.273320 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:57.273589 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 09:08:57.273997 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:08:57.275542 13951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:57.407164 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13825
I20260806 09:08:57.426327 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 13458
I20260806 09:08:57.427688 13521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:08:57.546025 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 13458
2026-08-06T09:08:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4961 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T09:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:08:57Z Disabled control of system clock
I20260806 09:08:57.654002 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38957
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:08:58.030445 14011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:58.030926 14011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:58.031111 14011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:58.039160 14011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:08:58.039295 14011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:58.039371 14011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:58.039462 14011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:08:58.039557 14011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:08:58.049659 14011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:38957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:58.051167 14011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:58.053212 14011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:58.066928 14017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:58.067019 14019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:58.066995 14016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:58.068077 14011 server_base.cc:1034] running on GCE node
I20260806 09:08:58.068940 14011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:58.070533 14011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:58.071753 14011 hybrid_clock.cc:648] HybridClock initialized: now 1786007338071697 us; error 46 us; skew 500 ppm
I20260806 09:08:58.072158 14011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:08:58.074929 14011 webserver.cc:460] Webserver started at http://127.27.187.190:43407/ using document root <none> and password file <none>
I20260806 09:08:58.075742 14011 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:58.075866 14011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:58.076234 14011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:58.079614 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "52385e8262ec43658ac4e6bee931e697"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "8e08c7140ad683d2f174cd84e374ea6e"
server_key_iv: "e1fbf879682f250bc2c884aab38a0053"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.080534 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "52385e8262ec43658ac4e6bee931e697"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "8e08c7140ad683d2f174cd84e374ea6e"
server_key_iv: "e1fbf879682f250bc2c884aab38a0053"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.086584 14011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:08:58.091364 14025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:58.093276 14011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:08:58.093549 14011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "52385e8262ec43658ac4e6bee931e697"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "8e08c7140ad683d2f174cd84e374ea6e"
server_key_iv: "e1fbf879682f250bc2c884aab38a0053"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.093865 14011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:58.126570 14011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:58.127656 14011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:58.128003 14011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:58.149300 14011 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:38957
I20260806 09:08:58.149343 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:38957 every 8 connection(s)
I20260806 09:08:58.150893 14011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:08:58.155298 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:58.159170 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14011
I20260806 09:08:58.159386 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:08:58.160048 28398 external_mini_cluster.cc:1428] Setting key a422ed3e20fca9f8db5ee7aec95ec044
I20260806 09:08:58.169561 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: Bootstrap starting.
I20260806 09:08:58.173111 14079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:58.174453 14079 log.cc:824] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:58.175493 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:58.162524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.27.187.190:38957)
0806 09:08:58.163701 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:58.163716 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:58.163766 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:08:58.165581 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:58.165610 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:58.166235 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:58.166739 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:58.167398 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.168714 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.170275 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.171068 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.171310 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:58.172580 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:58.172589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:58.172604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:08:58.173214 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:58.173369 (+ 155us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:08:58.173408 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:58.173556 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:58.173793 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 09:08:58.174152 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":374,"threads_started":1}
I20260806 09:08:58.178046 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: No bootstrap required, opened a new log
I20260806 09:08:58.186398 14079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } }
I20260806 09:08:58.186919 14079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:58.187055 14079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52385e8262ec43658ac4e6bee931e697, State: Initialized, Role: FOLLOWER
I20260806 09:08:58.187758 14079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } }
I20260806 09:08:58.188057 14079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:08:58.188215 14079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:08:58.188401 14079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:08:58.190927 14079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } }
I20260806 09:08:58.191555 14079 leader_election.cc:302] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52385e8262ec43658ac4e6bee931e697; no voters:
I20260806 09:08:58.192302 14079 leader_election.cc:288] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:08:58.192472 14084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:08:58.193269 14084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [term 1 LEADER]: Becoming Leader. State: Replica: 52385e8262ec43658ac4e6bee931e697, State: Running, Role: LEADER
I20260806 09:08:58.193939 14084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } }
I20260806 09:08:58.194806 14079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:08:58.198272 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52385e8262ec43658ac4e6bee931e697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } } }
I20260806 09:08:58.198946 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:58.198979 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52385e8262ec43658ac4e6bee931e697. Latest consensus state: current_term: 1 leader_uuid: "52385e8262ec43658ac4e6bee931e697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52385e8262ec43658ac4e6bee931e697" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 38957 } } }
I20260806 09:08:58.199316 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697 [sys.catalog]: This master's current role is: LEADER
I20260806 09:08:58.201038 14093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:08:58.207476 14093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:08:58.214363 14101 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:08:58.219615 14093 catalog_manager.cc:1347] Generated new cluster ID: de27cfca07c847de8e0d16d948a333c0
I20260806 09:08:58.219764 14093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:08:58.232609 14093 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:08:58.234247 14093 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:08:58.241591 14093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: Generated new TSK 0
I20260806 09:08:58.242506 14093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:08:58.251017 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--builtin_ntp_servers=127.27.187.148:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:08:58.648316 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:58.648732 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:58.648906 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:58.657498 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:58.657694 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:58.657894 14103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:08:58.668902 14103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:58.670608 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:58.672770 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:58.687686 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:58.687738 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:58.687631 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:58.692598 14103 server_base.cc:1034] running on GCE node
I20260806 09:08:58.693444 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:58.694648 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:58.695962 14103 hybrid_clock.cc:648] HybridClock initialized: now 1786007338695902 us; error 52 us; skew 500 ppm
I20260806 09:08:58.696375 14103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:08:58.699241 14103 webserver.cc:460] Webserver started at http://127.27.187.129:38967/ using document root <none> and password file <none>
I20260806 09:08:58.700181 14103 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:58.700321 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:58.700770 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:58.704213 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "58d73b4b4b3f4f65af5adeb6599bec20"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "072854fc409e4d67d7155c8d4d3ac6f8"
server_key_iv: "2e444d1c50be803fff9cb4f951a7317f"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.705112 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "58d73b4b4b3f4f65af5adeb6599bec20"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "072854fc409e4d67d7155c8d4d3ac6f8"
server_key_iv: "2e444d1c50be803fff9cb4f951a7317f"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.711134 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:08:58.715137 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:58.716687 14103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:08:58.716905 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "58d73b4b4b3f4f65af5adeb6599bec20"
format_stamp: "Formatted at 2026-08-06 09:08:58 on dist-test-slave-8tdl"
server_key: "072854fc409e4d67d7155c8d4d3ac6f8"
server_key_iv: "2e444d1c50be803fff9cb4f951a7317f"
server_key_version: "encryptionkey@0"
I20260806 09:08:58.717147 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:58.744833 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:58.745808 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:58.746155 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:58.747395 14103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:58.749121 14103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:58.749222 14103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:58.749339 14103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:58.749430 14103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:58.784739 14231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43627 every 8 connection(s)
I20260806 09:08:58.784719 14103 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43627
I20260806 09:08:58.786497 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:08:58.792140 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14103
I20260806 09:08:58.792418 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:08:58.792948 28398 external_mini_cluster.cc:1428] Setting key 2d027ed66ab4674dfd3f76a76710ecd2
I20260806 09:08:58.796674 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--builtin_ntp_servers=127.27.187.148:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:08:58.802239 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:58.789605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:54147 (local address 127.27.187.190:38957)
0806 09:08:58.790161 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:58.790166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:58.791652 (+ 1486us) server_negotiation.cc:407] Connection header received
0806 09:08:58.793804 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:58.793810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:58.793915 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:58.794046 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:58.795390 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.796260 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.798070 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.798565 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.798778 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:58.800990 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:58.800999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:58.801007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:08:58.801036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:58.801091 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:08:58.801110 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:58.801207 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:58.801636 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 09:08:58.801798 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":267,"threads_started":1}
I20260806 09:08:58.804162 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:58.790320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38957 (local address 127.27.187.129:54147)
0806 09:08:58.791467 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:58.791513 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:58.793456 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:58.794315 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:58.794324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:58.794432 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:58.795174 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:58.795194 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.796450 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.796454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:58.797900 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:58.797916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:58.798894 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:58.798904 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:58.799340 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:58.800446 (+ 1106us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:08:58.800453 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:58.800527 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:08:58.800724 (+ 197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:58.801340 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:58.801356 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:58.801368 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:08:58.801589 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:08:58.802201 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":381,"threads_started":1}
I20260806 09:08:58.806353 14232 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38957
I20260806 09:08:58.806815 14232 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:58.807709 14232 heartbeater.cc:511] Master 127.27.187.190:38957 requested a full tablet report, sending...
I20260806 09:08:58.810204 14043 ts_manager.cc:194] Registered new tserver with Master: 58d73b4b4b3f4f65af5adeb6599bec20 (127.27.187.129:43627)
I20260806 09:08:58.811981 14043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.129:54147
W20260806 09:08:59.181979 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:59.182327 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:59.182466 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:59.190414 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:59.190557 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:59.190786 14236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:08:59.201598 14236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:59.203111 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:59.205054 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:59.219911 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:59.219751 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:59.220062 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:59.221232 14236 server_base.cc:1034] running on GCE node
I20260806 09:08:59.222101 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:59.223405 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:59.224692 14236 hybrid_clock.cc:648] HybridClock initialized: now 1786007339224629 us; error 50 us; skew 500 ppm
I20260806 09:08:59.225095 14236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:08:59.228091 14236 webserver.cc:460] Webserver started at http://127.27.187.130:34439/ using document root <none> and password file <none>
I20260806 09:08:59.228888 14236 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:59.228986 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:59.229367 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:59.232640 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "8842e87b53bf4476b4f73085f797209d"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "eb2ad6f9837ce047fda614091df59966"
server_key_iv: "5cdf954bd21929678ef663e88e096522"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.233551 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "8842e87b53bf4476b4f73085f797209d"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "eb2ad6f9837ce047fda614091df59966"
server_key_iv: "5cdf954bd21929678ef663e88e096522"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.239496 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:08:59.244050 14250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.245775 14236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 09:08:59.245934 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "8842e87b53bf4476b4f73085f797209d"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "eb2ad6f9837ce047fda614091df59966"
server_key_iv: "5cdf954bd21929678ef663e88e096522"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.246135 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:59.269547 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:59.270455 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:59.270818 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:59.271914 14236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:59.273851 14236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:59.273994 14236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.274114 14236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:59.274199 14236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.312976 14236 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:43057
I20260806 09:08:59.313045 14363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:43057 every 8 connection(s)
I20260806 09:08:59.314792 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:08:59.315363 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14236
I20260806 09:08:59.315637 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:08:59.316310 28398 external_mini_cluster.cc:1428] Setting key c100fcd3a956ca6dd78c3e2337dfb34c
I20260806 09:08:59.320050 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--builtin_ntp_servers=127.27.187.148:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:08:59.329528 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.317962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:43979 (local address 127.27.187.190:38957)
0806 09:08:59.318532 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.318538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.320018 (+ 1480us) server_negotiation.cc:407] Connection header received
0806 09:08:59.322504 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.322513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.322603 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.322780 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:59.323989 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.324760 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.326400 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.326929 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.327080 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.328608 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:59.328613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:59.328616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:08:59.328638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:59.328683 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:08:59.328695 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:59.328794 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.329033 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.329148 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":302,"threads_started":1}
I20260806 09:08:59.330752 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.318782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38957 (local address 127.27.187.130:43979)
0806 09:08:59.319856 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:59.319901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:59.322100 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:59.323031 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:59.323041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:59.323152 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:59.323784 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.323803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.324938 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.324942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.326240 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.326253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.327177 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.327185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.327437 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.328238 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:08:59.328241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:59.328285 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:08:59.328377 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:59.328795 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:59.328809 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:59.328820 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:08:59.329020 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 09:08:59.329467 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":361,"threads_started":1}
I20260806 09:08:59.332352 14364 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38957
I20260806 09:08:59.332686 14364 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:59.333534 14364 heartbeater.cc:511] Master 127.27.187.190:38957 requested a full tablet report, sending...
I20260806 09:08:59.335222 14043 ts_manager.cc:194] Registered new tserver with Master: 8842e87b53bf4476b4f73085f797209d (127.27.187.130:43057)
I20260806 09:08:59.336081 14043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.130:43979
W20260806 09:08:59.694746 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:08:59.695111 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:08:59.695267 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:08:59.703488 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:08:59.703636 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:08:59.703789 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:08:59.714134 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:08:59.715595 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:08:59.717610 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:08:59.734781 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:59.734714 14377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:08:59.734817 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:08:59.735910 14368 server_base.cc:1034] running on GCE node
I20260806 09:08:59.736713 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:08:59.737890 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:08:59.739123 14368 hybrid_clock.cc:648] HybridClock initialized: now 1786007339739068 us; error 60 us; skew 500 ppm
I20260806 09:08:59.739652 14368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:08:59.742871 14368 webserver.cc:460] Webserver started at http://127.27.187.131:38333/ using document root <none> and password file <none>
I20260806 09:08:59.743836 14368 fs_manager.cc:359] Metadata directory not provided
I20260806 09:08:59.743937 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:08:59.744217 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:08:59.747375 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "0f1952a178234c92939b8a9a15518b71"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "9b98beab47f73a7a0f2a34ed907ad26c"
server_key_iv: "f05db130b5c467a8e5049f64bbeec80f"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.748242 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "0f1952a178234c92939b8a9a15518b71"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "9b98beab47f73a7a0f2a34ed907ad26c"
server_key_iv: "f05db130b5c467a8e5049f64bbeec80f"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.754325 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:08:59.758764 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.760695 14368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:08:59.760929 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "0f1952a178234c92939b8a9a15518b71"
format_stamp: "Formatted at 2026-08-06 09:08:59 on dist-test-slave-8tdl"
server_key: "9b98beab47f73a7a0f2a34ed907ad26c"
server_key_iv: "f05db130b5c467a8e5049f64bbeec80f"
server_key_version: "encryptionkey@0"
I20260806 09:08:59.761199 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:08:59.788733 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:08:59.789634 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:08:59.789984 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:08:59.791168 14368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:08:59.792824 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:08:59.792953 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.793090 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:08:59.793190 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:08:59.815587 14232 heartbeater.cc:503] Master 127.27.187.190:38957 was elected leader, sending a full tablet report...
I20260806 09:08:59.831988 14368 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:34589
I20260806 09:08:59.832075 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:34589 every 8 connection(s)
I20260806 09:08:59.833463 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:08:59.835785 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14368
I20260806 09:08:59.836004 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:08:59.836572 28398 external_mini_cluster.cc:1428] Setting key b1b294816ddd105025001ec7ba50f846
I20260806 09:08:59.848519 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.836495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:59841 (local address 127.27.187.190:38957)
0806 09:08:59.837213 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.837220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.838890 (+ 1670us) server_negotiation.cc:407] Connection header received
0806 09:08:59.841381 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.841386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.841464 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.841569 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:59.842931 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.843818 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.845545 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.845978 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.846083 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.847651 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:59.847656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:59.847659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:08:59.847682 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:59.847744 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:08:59.847758 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:59.847836 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.848086 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.848195 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":383,"threads_started":1}
I20260806 09:08:59.849644 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.837206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:38957 (local address 127.27.187.131:59841)
0806 09:08:59.838614 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:59.838736 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:59.841068 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:59.841908 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:59.841919 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:59.842033 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:08:59.842657 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.842752 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.843992 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.843996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.845376 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.845391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.846195 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.846200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.846414 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.847283 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:08:59.847286 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:08:59.847332 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:08:59.847429 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:08:59.847854 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:08:59.847862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:08:59.847868 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:08:59.848022 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:08:59.848381 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":440,"threads_started":1}
I20260806 09:08:59.851925 14499 heartbeater.cc:347] Connected to a master server at 127.27.187.190:38957
I20260806 09:08:59.852340 14499 heartbeater.cc:464] Registering TS with master...
I20260806 09:08:59.853327 14499 heartbeater.cc:511] Master 127.27.187.190:38957 requested a full tablet report, sending...
I20260806 09:08:59.855086 14043 ts_manager.cc:194] Registered new tserver with Master: 0f1952a178234c92939b8a9a15518b71 (127.27.187.131:34589)
I20260806 09:08:59.855985 14043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.131:59841
I20260806 09:08:59.866935 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:08:59.878579 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.871848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.27.187.190:38957)
0806 09:08:59.872182 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.872188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.872454 (+ 266us) server_negotiation.cc:407] Connection header received
0806 09:08:59.872678 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.872683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.872760 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.872862 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:08:59.873471 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.874239 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.875962 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.876460 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.876580 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.877826 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:08:59.877832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:08:59.877835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:08:59.877858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:08:59.877896 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:08:59.877908 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:08:59.877990 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.878211 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.878315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:08:59.883507 14043 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:08:59.891731 14043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:08:59.934898 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.907168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:43057 (local address 127.0.0.1:51428)
0806 09:08:59.908101 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:59.908146 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:59.908485 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:59.912047 (+ 3562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:59.912060 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:59.912113 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:59.912670 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.912688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.918126 (+ 5438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.918137 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.924183 (+ 6046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.924234 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.931006 (+ 6772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.931017 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.933508 (+ 2491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.933657 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 09:08:59.933923 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 09:08:59.934299 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":434,"threads_started":1}
I20260806 09:08:59.935856 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.907249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.27.187.130:43057)
0806 09:08:59.907976 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.907986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.908344 (+ 358us) server_negotiation.cc:407] Connection header received
0806 09:08:59.908817 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.908830 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.909142 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.911520 (+ 2378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:59.913001 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.917779 (+ 4778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.926489 (+ 8710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.930081 (+ 3592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.930388 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.930742 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.935318 (+ 4576us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.935549 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":350,"threads_started":1}
I20260806 09:08:59.940761 14298 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 (DEFAULT_TABLE table=test-table [id=657d7422622c49be892ede390cc6262c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:59.942528 14298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3040416b624d8dae17bf67dbe3d9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:59.944504 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.908664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:34589 (local address 127.0.0.1:39620)
0806 09:08:59.934946 (+ 26282us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:59.934961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:59.936222 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:59.937357 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:59.937362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:59.937390 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:59.937738 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.937748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.939372 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.939382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.941584 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.941601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.943481 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.943489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.943682 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.943726 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 09:08:59.943862 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:08:59.944034 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26170,"mutex_wait_us":1,"thread_start_us":25359,"threads_started":1}
I20260806 09:08:59.944593 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.909316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.27.187.131:34589)
0806 09:08:59.932727 (+ 23411us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.932742 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.935110 (+ 2368us) server_negotiation.cc:407] Connection header received
0806 09:08:59.936487 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.936499 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.936753 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.936950 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:59.937950 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.939169 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.941802 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.943224 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.943481 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.943726 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.943908 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.944090 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23123,"thread_start_us":23444,"threads_started":1}
I20260806 09:08:59.944881 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.908566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41784)
0806 09:08:59.933639 (+ 25073us) negotiation.cc:107] Waiting for socket to connect
0806 09:08:59.933676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:08:59.933870 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:08:59.936645 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:08:59.936649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:08:59.936678 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:08:59.937005 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.937015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.938956 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.938961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.941437 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:08:59.941459 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.943989 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.943996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:08:59.944170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.944216 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:08:59.944365 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 09:08:59.944531 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24864,"mutex_wait_us":37,"thread_start_us":26104,"threads_started":1}
I20260806 09:08:59.945592 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:08:59.908991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.27.187.129:43627)
0806 09:08:59.935629 (+ 26638us) server_negotiation.cc:206] Beginning negotiation
0806 09:08:59.935636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:08:59.935662 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:08:59.935753 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:08:59.935760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:08:59.936428 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:08:59.936665 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:08:59.937177 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.938788 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.941730 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:08:59.943391 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:08:59.943617 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:08:59.943827 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 09:08:59.945030 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0806 09:08:59.945225 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26475,"thread_start_us":26352,"threads_started":1}
I20260806 09:08:59.949517 14433 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 (DEFAULT_TABLE table=test-table [id=657d7422622c49be892ede390cc6262c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:59.950194 14165 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 (DEFAULT_TABLE table=test-table [id=657d7422622c49be892ede390cc6262c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:08:59.951313 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3040416b624d8dae17bf67dbe3d9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:59.951772 14165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3040416b624d8dae17bf67dbe3d9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:08:59.959450 14518 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Bootstrap starting.
I20260806 09:08:59.963891 14518 tablet_bootstrap.cc:654] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:59.965888 14519 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Bootstrap starting.
I20260806 09:08:59.966796 14518 log.cc:824] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:59.968453 14520 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Bootstrap starting.
I20260806 09:08:59.970916 14519 tablet_bootstrap.cc:654] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:59.971925 14518 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: No bootstrap required, opened a new log
I20260806 09:08:59.972256 14518 ts_tablet_manager.cc:1397] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260806 09:08:59.972306 14519 log.cc:824] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:59.972627 14520 tablet_bootstrap.cc:654] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Neither blocks nor log segments found. Creating new log.
I20260806 09:08:59.973814 14520 log.cc:824] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Log is configured to *not* fsync() on all Append() calls
I20260806 09:08:59.976476 14519 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: No bootstrap required, opened a new log
I20260806 09:08:59.976809 14519 ts_tablet_manager.cc:1397] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 09:08:59.977378 14520 tablet_bootstrap.cc:492] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: No bootstrap required, opened a new log
I20260806 09:08:59.977670 14520 ts_tablet_manager.cc:1397] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:08:59.981962 14518 raft_consensus.cc:348] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.982414 14518 raft_consensus.cc:374] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:59.982519 14518 raft_consensus.cc:729] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8842e87b53bf4476b4f73085f797209d, State: Initialized, Role: FOLLOWER
I20260806 09:08:59.983326 14518 consensus_queue.cc:260] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.984963 14364 heartbeater.cc:503] Master 127.27.187.190:38957 was elected leader, sending a full tablet report...
I20260806 09:08:59.987262 14518 ts_tablet_manager.cc:1428] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260806 09:08:59.988582 14519 raft_consensus.cc:348] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.989120 14519 raft_consensus.cc:374] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:59.989234 14519 raft_consensus.cc:729] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1952a178234c92939b8a9a15518b71, State: Initialized, Role: FOLLOWER
I20260806 09:08:59.989948 14519 consensus_queue.cc:260] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.989938 14520 raft_consensus.cc:348] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.990434 14520 raft_consensus.cc:374] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:08:59.990581 14520 raft_consensus.cc:729] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d73b4b4b3f4f65af5adeb6599bec20, State: Initialized, Role: FOLLOWER
I20260806 09:08:59.991608 14499 heartbeater.cc:503] Master 127.27.187.190:38957 was elected leader, sending a full tablet report...
I20260806 09:08:59.991497 14520 consensus_queue.cc:260] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:08:59.992143 14519 ts_tablet_manager.cc:1428] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Time spent starting tablet: real 0.015s user 0.010s sys 0.007s
I20260806 09:08:59.994025 14520 ts_tablet_manager.cc:1428] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260806 09:09:00.039575 14233 tablet.cc:2369] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:00.067462 14365 tablet.cc:2369] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:00.085485 14500 tablet.cc:2369] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:09:00.197404 14526 raft_consensus.cc:482] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:09:00.197835 14526 raft_consensus.cc:504] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:09:00.200209 14526 leader_election.cc:288] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8842e87b53bf4476b4f73085f797209d (127.27.187.130:43057), 0f1952a178234c92939b8a9a15518b71 (127.27.187.131:34589)
I20260806 09:09:00.207862 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.200963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:47195 (local address 127.27.187.131:34589)
0806 09:09:00.201261 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.201271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.202050 (+ 779us) server_negotiation.cc:407] Connection header received
0806 09:09:00.202258 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.202263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.202325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.202433 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:00.203133 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.203847 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.205891 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.206825 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.207017 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.207128 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.207458 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.207600 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:09:00.207859 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.200726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:55895 (local address 127.27.187.130:43057)
0806 09:09:00.200984 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.200994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.201764 (+ 770us) server_negotiation.cc:407] Connection header received
0806 09:09:00.202026 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.202030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.202104 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.202224 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:00.202906 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.203647 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.205962 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.206822 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.207007 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.207089 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.207432 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.207571 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:09:00.208132 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.200650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:34589 (local address 127.27.187.129:47195)
0806 09:09:00.201907 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.201922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:00.202093 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:00.202455 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:00.202459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:00.202489 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:00.202941 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:00.202951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.204010 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.204015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:00.205643 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:00.205659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.207108 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.207117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:00.207286 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.207310 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:09:00.207424 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 09:09:00.207572 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":5,"thread_start_us":509,"threads_started":1}
I20260806 09:09:00.208576 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.200600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:43057 (local address 127.27.187.129:55895)
0806 09:09:00.201609 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.201649 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:00.201848 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:00.202240 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:00.202245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:00.202272 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:00.202737 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:00.202756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.203803 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.203809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:00.205697 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:00.205716 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.207055 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.207062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:00.207247 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.207283 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:09:00.207400 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:09:00.207551 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":133,"thread_start_us":912,"threads_started":1}
I20260806 09:09:00.209503 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3040416b624d8dae17bf67dbe3d9f0" candidate_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1952a178234c92939b8a9a15518b71" is_pre_election: true
I20260806 09:09:00.209777 14318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3040416b624d8dae17bf67dbe3d9f0" candidate_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8842e87b53bf4476b4f73085f797209d" is_pre_election: true
I20260806 09:09:00.210068 14453 raft_consensus.cc:2454] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d73b4b4b3f4f65af5adeb6599bec20 in term 0.
I20260806 09:09:00.210391 14318 raft_consensus.cc:2454] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d73b4b4b3f4f65af5adeb6599bec20 in term 0.
I20260806 09:09:00.210999 14121 leader_election.cc:302] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1952a178234c92939b8a9a15518b71, 58d73b4b4b3f4f65af5adeb6599bec20; no voters:
I20260806 09:09:00.211589 14526 raft_consensus.cc:2781] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:09:00.211784 14526 raft_consensus.cc:482] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:09:00.211920 14526 raft_consensus.cc:3037] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:00.215003 14526 raft_consensus.cc:504] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:09:00.215973 14526 leader_election.cc:288] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [CANDIDATE]: Term 1 election: Requested vote from peers 8842e87b53bf4476b4f73085f797209d (127.27.187.130:43057), 0f1952a178234c92939b8a9a15518b71 (127.27.187.131:34589)
I20260806 09:09:00.216476 14318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3040416b624d8dae17bf67dbe3d9f0" candidate_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8842e87b53bf4476b4f73085f797209d"
I20260806 09:09:00.216756 14318 raft_consensus.cc:3037] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:00.216681 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3040416b624d8dae17bf67dbe3d9f0" candidate_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1952a178234c92939b8a9a15518b71"
I20260806 09:09:00.216992 14453 raft_consensus.cc:3037] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:00.220733 14318 raft_consensus.cc:2454] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d73b4b4b3f4f65af5adeb6599bec20 in term 1.
I20260806 09:09:00.220742 14453 raft_consensus.cc:2454] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d73b4b4b3f4f65af5adeb6599bec20 in term 1.
I20260806 09:09:00.221398 14120 leader_election.cc:302] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d73b4b4b3f4f65af5adeb6599bec20, 8842e87b53bf4476b4f73085f797209d; no voters:
I20260806 09:09:00.221903 14526 raft_consensus.cc:2781] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:00.222502 14526 raft_consensus.cc:686] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 1 LEADER]: Becoming Leader. State: Replica: 58d73b4b4b3f4f65af5adeb6599bec20, State: Running, Role: LEADER
I20260806 09:09:00.223299 14526 consensus_queue.cc:237] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } }
I20260806 09:09:00.228900 14041 catalog_manager.cc:5495] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d73b4b4b3f4f65af5adeb6599bec20 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d73b4b4b3f4f65af5adeb6599bec20" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 } health_report { overall_health: UNKNOWN } } }
I20260806 09:09:00.300750 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.293936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.27.187.129:43627)
0806 09:09:00.294123 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.294128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.294148 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:09:00.294360 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.294365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.294507 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.294733 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:09:00.295096 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.296035 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.297617 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.298394 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.298521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.299225 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:09:00.300041 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:09:00.300161 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.300332 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.300409 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 09:09:00.310040 14318 raft_consensus.cc:1261] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 1 FOLLOWER]: Refusing update from remote peer 58d73b4b4b3f4f65af5adeb6599bec20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:00.310042 14453 raft_consensus.cc:1261] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 1 FOLLOWER]: Refusing update from remote peer 58d73b4b4b3f4f65af5adeb6599bec20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:00.311136 14529 consensus_queue.cc:1035] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8842e87b53bf4476b4f73085f797209d" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:00.311532 14526 consensus_queue.cc:1035] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1952a178234c92939b8a9a15518b71" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:00.330969 14533 mvcc.cc:205] Tried to move back new op lower bound from 7315486065899528192 to 7315486065559142400. Current Snapshot: MvccSnapshot[applied={T|T < 7315486065899528192}]
I20260806 09:09:00.343389 14534 mvcc.cc:205] Tried to move back new op lower bound from 7315486065899528192 to 7315486065559142400. Current Snapshot: MvccSnapshot[applied={T|T < 7315486065899528192}]
I20260806 09:09:00.354266 14043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:00.354579 14043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:00.358650 14043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52385e8262ec43658ac4e6bee931e697: Sending DeleteTablet for 3 replicas of tablet 5b3040416b624d8dae17bf67dbe3d9f0
I20260806 09:09:00.360144 14298 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:00 UTC) from {username='slave'} at 127.0.0.1:51428
I20260806 09:09:00.360217 14165 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:00 UTC) from {username='slave'} at 127.0.0.1:41784
I20260806 09:09:00.360635 14433 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3040416b624d8dae17bf67dbe3d9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:00 UTC) from {username='slave'} at 127.0.0.1:39620
I20260806 09:09:00.361568 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14103
I20260806 09:09:00.382884 14543 tablet_replica.cc:330] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: stopping tablet replica
I20260806 09:09:00.383638 14543 raft_consensus.cc:2229] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:00.384426 14543 raft_consensus.cc:2258] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:00.385442 14544 tablet_replica.cc:330] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: stopping tablet replica
I20260806 09:09:00.386062 14544 raft_consensus.cc:2229] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:00.388846 14543 ts_tablet_manager.cc:1905] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:00.389637 14542 tablet_replica.cc:330] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: stopping tablet replica
I20260806 09:09:00.389986 14544 raft_consensus.cc:2258] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:00.390098 14542 raft_consensus.cc:2229] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:09:00.390841 14542 raft_consensus.cc:2258] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:00.393070 14542 ts_tablet_manager.cc:1905] T 5b3040416b624d8dae17bf67dbe3d9f0 P 58d73b4b4b3f4f65af5adeb6599bec20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:00.393146 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.363982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.27.187.129:43627)
0806 09:09:00.364779 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.364790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.374924 (+ 10134us) server_negotiation.cc:407] Connection header received
0806 09:09:00.375639 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.375649 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.375817 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.375931 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:00.376990 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.378970 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.386746 (+ 7776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.387882 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.388089 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.391003 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:00.391016 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:00.391034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:00.391982 (+ 948us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:00.392080 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:00.392115 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:00.392232 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.392578 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.392713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570}
I20260806 09:09:00.394529 14227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:00.396361 14544 ts_tablet_manager.cc:1905] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:00.400511 14543 ts_tablet_manager.cc:1918] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:00.400761 14543 log.cc:1196] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/5b3040416b624d8dae17bf67dbe3d9f0
I20260806 09:09:00.401485 14543 ts_tablet_manager.cc:1939] T 5b3040416b624d8dae17bf67dbe3d9f0 P 8842e87b53bf4476b4f73085f797209d: Deleting consensus metadata
I20260806 09:09:00.404033 14028 catalog_manager.cc:4841] TS 8842e87b53bf4476b4f73085f797209d (127.27.187.130:43057): tablet 5b3040416b624d8dae17bf67dbe3d9f0 (table test-table [id=657d7422622c49be892ede390cc6262c]) successfully deleted
I20260806 09:09:00.405055 14544 ts_tablet_manager.cc:1918] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:00.405275 14544 log.cc:1196] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/5b3040416b624d8dae17bf67dbe3d9f0
I20260806 09:09:00.406136 14544 ts_tablet_manager.cc:1939] T 5b3040416b624d8dae17bf67dbe3d9f0 P 0f1952a178234c92939b8a9a15518b71: Deleting consensus metadata
I20260806 09:09:00.408438 14029 catalog_manager.cc:4841] TS 0f1952a178234c92939b8a9a15518b71 (127.27.187.131:34589): tablet 5b3040416b624d8dae17bf67dbe3d9f0 (table test-table [id=657d7422622c49be892ede390cc6262c]) successfully deleted
I20260806 09:09:00.505602 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14103
W20260806 09:09:00.525743 14028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:43627 (error 108)
I20260806 09:09:00.526621 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14236
I20260806 09:09:00.528391 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.527488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41810)
0806 09:09:00.528007 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.528184 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
W20260806 09:09:00.528657 14028 catalog_manager.cc:4584] TS 58d73b4b4b3f4f65af5adeb6599bec20 (127.27.187.129:43627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b3040416b624d8dae17bf67dbe3d9f0: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
I20260806 09:09:00.534464 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.527558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.27.187.130:43057)
0806 09:09:00.527803 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.527809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.527832 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:09:00.527995 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.527999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.528090 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.528177 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:00.528727 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.529539 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.531217 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.531668 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.531822 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.532985 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:00.532994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:00.533010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:00.533636 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:00.533686 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:00.533713 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:00.533845 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.533988 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.534108 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:09:00.535270 14359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:00.582450 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.581833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41812)
0806 09:09:00.582132 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.582256 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260806 09:09:00.583861 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.583365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41822)
0806 09:09:00.583560 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.583665 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:09:00.633059 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14236
I20260806 09:09:00.644788 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.644278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41838)
0806 09:09:00.644507 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.644619 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:09:00.645885 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.645491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41844)
0806 09:09:00.645653 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.645738 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 09:09:00.652984 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14368
I20260806 09:09:00.660964 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.653941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.27.187.131:34589)
0806 09:09:00.654121 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:00.654125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:00.654145 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:09:00.654207 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:00.654210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:00.654299 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:00.654398 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:00.654995 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.655781 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.657350 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:00.657785 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:00.657888 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:00.659499 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:00.659510 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:00.659528 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:00.660162 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:00.660217 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:00.660239 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:00.660349 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:00.660543 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 09:09:00.660687 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:09:00.661839 14494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:00.726421 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.725831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41854)
0806 09:09:00.726101 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.726233 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 09:09:00.727931 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:00.727448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43627 (local address 127.0.0.1:41866)
0806 09:09:00.727681 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:00.727775 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 09:09:00.755596 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14368
I20260806 09:09:00.775296 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14011
I20260806 09:09:00.776547 14074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:00.894032 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14011
2026-08-06T09:09:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3341 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:01 dist-test-slave-8tdl krb5kdc[14554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:01 dist-test-slave-8tdl krb5kdc[14554](info): set up 2 sockets
Aug 06 09:09:01 dist-test-slave-8tdl krb5kdc[14554](info): commencing operation
krb5kdc: starting...
W20260806 09:09:02.065516 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:02 dist-test-slave-8tdl krb5kdc[14554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:02.415076 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:35035
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:02.793790 14567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:02.794133 14567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:02.794225 14567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:02.802044 14567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:02.802191 14567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:02.802241 14567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:02.802295 14567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:02.802341 14567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:02.813051 14567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:02.814463 14567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:02.815897 14567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:02.828896 14573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:02.828897 14572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:02.828933 14575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:02.830235 14567 server_base.cc:1034] running on GCE node
I20260806 09:09:02.831251 14567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:02.832957 14567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:02.834219 14567 hybrid_clock.cc:648] HybridClock initialized: now 1786007342834158 us; error 43 us; skew 500 ppm
I20260806 09:09:02.834721 14567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:02 dist-test-slave-8tdl krb5kdc[14554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:02.843328 14567 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:02.845396 14567 webserver.cc:460] Webserver started at http://127.27.187.190:33247/ using document root <none> and password file <none>
I20260806 09:09:02.846258 14567 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:02.846436 14567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:02.846854 14567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:02.850337 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "5fe2a929027c419e9c2aa2a3ff58e501"
format_stamp: "Formatted at 2026-08-06 09:09:02 on dist-test-slave-8tdl"
server_key: "df3895c720b358f98c4b99b438edf26b"
server_key_iv: "b58f3b1dd58462ec6ce5832a31c97b31"
server_key_version: "encryptionkey@0"
I20260806 09:09:02.851263 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "5fe2a929027c419e9c2aa2a3ff58e501"
format_stamp: "Formatted at 2026-08-06 09:09:02 on dist-test-slave-8tdl"
server_key: "df3895c720b358f98c4b99b438edf26b"
server_key_iv: "b58f3b1dd58462ec6ce5832a31c97b31"
server_key_version: "encryptionkey@0"
I20260806 09:09:02.857656 14567 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:09:02.862032 14582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:02.863907 14567 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:09:02.864162 14567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "5fe2a929027c419e9c2aa2a3ff58e501"
format_stamp: "Formatted at 2026-08-06 09:09:02 on dist-test-slave-8tdl"
server_key: "df3895c720b358f98c4b99b438edf26b"
server_key_iv: "b58f3b1dd58462ec6ce5832a31c97b31"
server_key_version: "encryptionkey@0"
I20260806 09:09:02.864413 14567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:02.881867 14567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:02.887038 14567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:02.887387 14567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:02.908862 14567 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:35035
I20260806 09:09:02.908893 14633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:35035 every 8 connection(s)
I20260806 09:09:02.910523 14567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:02.914909 14634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:02.920459 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14567
I20260806 09:09:02.920712 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:02.921294 28398 external_mini_cluster.cc:1428] Setting key f512bfed0a9972d3a661b39e12c7d841
I20260806 09:09:02.927872 14634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: Bootstrap starting.
I20260806 09:09:02.932008 14634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:02.933394 14634 log.cc:824] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:02.937772 14634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: No bootstrap required, opened a new log
Aug 06 09:09:02 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:02.948225 14634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } }
I20260806 09:09:02.948704 14634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:02.948838 14634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe2a929027c419e9c2aa2a3ff58e501, State: Initialized, Role: FOLLOWER
I20260806 09:09:02.949522 14634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } }
I20260806 09:09:02.949839 14634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:02.950001 14634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:02.950176 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:02.923514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.27.187.190:35035)
0806 09:09:02.924494 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:02.924507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:02.924557 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:09:02.926477 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:02.926504 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:02.927074 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:02.927445 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:02.929487 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:02.931007 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:02.933072 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:02.934105 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:02.934358 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:02.940521 (+ 6163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:02.940533 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:02.940553 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:02.940607 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:02.944438 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:02.945501 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:02.945509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:02.945517 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:02.945741 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:02.946185 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:02.946188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:02.946190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:02.946914 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:02.947165 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:02.947893 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 09:09:02.948469 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":478,"threads_started":1}
I20260806 09:09:02.950196 14634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:02.953392 14634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } }
I20260806 09:09:02.954109 14634 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe2a929027c419e9c2aa2a3ff58e501; no voters:
I20260806 09:09:02.954823 14634 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:02.955070 14639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:02.955861 14639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe2a929027c419e9c2aa2a3ff58e501, State: Running, Role: LEADER
I20260806 09:09:02.956537 14639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } }
I20260806 09:09:02.957136 14634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:02.960770 14641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe2a929027c419e9c2aa2a3ff58e501. Latest consensus state: current_term: 1 leader_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } } }
I20260806 09:09:02.960744 14640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe2a929027c419e9c2aa2a3ff58e501" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 35035 } } }
I20260806 09:09:02.961359 14641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:02.961361 14640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:02.964015 14646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:02.971297 14646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:02.982476 14646 catalog_manager.cc:1347] Generated new cluster ID: e64b7a1b3c38461db8bb0a2e1492e79d
I20260806 09:09:02.982623 14646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:03.004458 14646 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:03.006446 14646 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:03.021093 14646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: Generated new TSK 0
I20260806 09:09:03.022294 14646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:09:03.104385 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:03.486483 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:03.486881 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:03.486987 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:03.494804 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:03.494935 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:03.495059 14660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:09:03.505292 14660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:03.506665 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:03.508235 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:03.523229 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:03.523207 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:03.523207 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:03.524014 14660 server_base.cc:1034] running on GCE node
I20260806 09:09:03.524739 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:03.525753 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:03.526980 14660 hybrid_clock.cc:648] HybridClock initialized: now 1786007343526891 us; error 72 us; skew 500 ppm
I20260806 09:09:03.527372 14660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:03 dist-test-slave-8tdl krb5kdc[14554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:03.530910 14660 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:09:03.532706 14660 webserver.cc:460] Webserver started at http://127.27.187.129:37277/ using document root <none> and password file <none>
I20260806 09:09:03.533423 14660 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:03.533535 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:03.533830 14660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:03.537067 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "af249ae4a4b44af892213700a4a17cab"
format_stamp: "Formatted at 2026-08-06 09:09:03 on dist-test-slave-8tdl"
server_key: "34c8e71a3318f56f2abf858f7bdbac97"
server_key_iv: "b0bd41c4062e76d1e383ddd8a1cde125"
server_key_version: "encryptionkey@0"
I20260806 09:09:03.537842 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "af249ae4a4b44af892213700a4a17cab"
format_stamp: "Formatted at 2026-08-06 09:09:03 on dist-test-slave-8tdl"
server_key: "34c8e71a3318f56f2abf858f7bdbac97"
server_key_iv: "b0bd41c4062e76d1e383ddd8a1cde125"
server_key_version: "encryptionkey@0"
I20260806 09:09:03.544170 14660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260806 09:09:03.548137 14675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:03.549774 14660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:09:03.549938 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "af249ae4a4b44af892213700a4a17cab"
format_stamp: "Formatted at 2026-08-06 09:09:03 on dist-test-slave-8tdl"
server_key: "34c8e71a3318f56f2abf858f7bdbac97"
server_key_iv: "b0bd41c4062e76d1e383ddd8a1cde125"
server_key_version: "encryptionkey@0"
I20260806 09:09:03.550136 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:03.589059 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:03.595433 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:03.595721 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:03.596796 14660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:03.598311 14660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:03.598390 14660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:03.598486 14660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:03.598522 14660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:03.635452 14660 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:39859
I20260806 09:09:03.635471 14789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:39859 every 8 connection(s)
I20260806 09:09:03.636931 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:09:03.643287 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14660
I20260806 09:09:03.643512 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:09:03.644127 28398 external_mini_cluster.cc:1428] Setting key 1ee2cd301932df450095afa551f186bd
Aug 06 09:09:03 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:03.664325 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:03.639968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:55815 (local address 127.27.187.190:35035)
0806 09:09:03.640515 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:03.640520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:03.642774 (+ 2254us) server_negotiation.cc:407] Connection header received
0806 09:09:03.645038 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:03.645045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:03.645155 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:03.645288 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:03.647430 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:03.648252 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:03.651195 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:03.651658 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:03.651788 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:03.656782 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:03.656788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:03.656791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:03.656834 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:03.659835 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:03.661034 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:03.661039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:03.661041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:03.661192 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:03.661888 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:03.661896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:03.661902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:03.662390 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:03.662570 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:03.663513 (+ 943us) server_negotiation.cc:299] Negotiation successful
0806 09:09:03.663802 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":299,"threads_started":1}
I20260806 09:09:03.665318 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:03.640718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35035 (local address 127.27.187.129:55815)
0806 09:09:03.641811 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:03.641864 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:03.644612 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:03.645594 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:03.645604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:03.646262 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:03.647194 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:03.647226 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:03.648468 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:03.648476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:03.650970 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:03.650994 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:03.651946 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:03.651956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:03.652343 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:03.653221 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:03.653268 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:03.656453 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:03.660023 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:03.660034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:03.660065 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:03.660846 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:03.661398 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:03.661406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:03.661412 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:03.661716 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:03.662554 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:03.662561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:03.663041 (+ 480us) client_negotiation.cc:769] Sending connection context
0806 09:09:03.663335 (+ 294us) client_negotiation.cc:240] Negotiation successful
0806 09:09:03.663791 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":481,"threads_started":1}
I20260806 09:09:03.667276 14790 heartbeater.cc:347] Connected to a master server at 127.27.187.190:35035
I20260806 09:09:03.667620 14790 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:03.668484 14790 heartbeater.cc:511] Master 127.27.187.190:35035 requested a full tablet report, sending...
I20260806 09:09:03.671299 14599 ts_manager.cc:194] Registered new tserver with Master: af249ae4a4b44af892213700a4a17cab (127.27.187.129:39859)
I20260806 09:09:03.673627 14599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:55815
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:09:03.724720 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:04.108812 14797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:04.109196 14797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:04.109373 14797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:04.116936 14797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:04.117156 14797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:04.117376 14797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:09:04.128058 14797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:04.129568 14797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:04.131072 14797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:04.146018 14803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:04.146229 14805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:04.146219 14802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:04.147439 14797 server_base.cc:1034] running on GCE node
I20260806 09:09:04.148126 14797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:04.149317 14797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:04.150568 14797 hybrid_clock.cc:648] HybridClock initialized: now 1786007344150509 us; error 61 us; skew 500 ppm
I20260806 09:09:04.151011 14797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:04 dist-test-slave-8tdl krb5kdc[14554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:04.154721 14797 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:09:04.156620 14797 webserver.cc:460] Webserver started at http://127.27.187.130:38217/ using document root <none> and password file <none>
I20260806 09:09:04.157451 14797 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:04.157588 14797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:04.158041 14797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:04.161363 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "61cf626de2ef4fa6b34b14e628d4481f"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "7085ffe9a4e85865575351c621568bb6"
server_key_iv: "3f9da9830fd7dfff7ee0baf2bc04dae1"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.162240 14797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "61cf626de2ef4fa6b34b14e628d4481f"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "7085ffe9a4e85865575351c621568bb6"
server_key_iv: "3f9da9830fd7dfff7ee0baf2bc04dae1"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.168744 14797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 09:09:04.173018 14812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.174662 14797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:09:04.175004 14797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "61cf626de2ef4fa6b34b14e628d4481f"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "7085ffe9a4e85865575351c621568bb6"
server_key_iv: "3f9da9830fd7dfff7ee0baf2bc04dae1"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.175244 14797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:04.192373 14797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:04.197069 14797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:04.197413 14797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:04.198535 14797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:04.200107 14797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:04.200225 14797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.200392 14797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:04.200472 14797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.237231 14797 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:36047
I20260806 09:09:04.237344 14924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:36047 every 8 connection(s)
I20260806 09:09:04.238659 14797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:09:04.247596 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14797
I20260806 09:09:04.247813 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:09:04.248332 28398 external_mini_cluster.cc:1428] Setting key 5aafd5c38ec2724f7d797bec0b7ca19c
Aug 06 09:09:04 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:04.264006 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.241660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:56619 (local address 127.27.187.190:35035)
0806 09:09:04.242221 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.242226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.243208 (+ 982us) server_negotiation.cc:407] Connection header received
0806 09:09:04.245388 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.245394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.245494 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.245627 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:04.247507 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.248503 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.250821 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.251288 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.251393 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.256434 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:04.256439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:04.256442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:04.256483 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:04.260028 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.261055 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.261060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.261063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.261224 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.261758 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.261763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.261765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.262090 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:04.262288 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.263237 (+ 949us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.263557 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260806 09:09:04.265339 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.242222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35035 (local address 127.27.187.130:56619)
0806 09:09:04.243020 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:04.243066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:04.245048 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:04.245895 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:04.245904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:04.246498 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:04.247212 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.247240 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.248693 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.248699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.250606 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.250630 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.251549 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.251560 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.251946 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.252841 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:04.252880 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:04.256045 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:04.260252 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:04.260265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:04.260295 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:04.260878 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:04.261357 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:04.261362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:04.261367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:04.261609 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:04.262254 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:04.262264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:04.262767 (+ 503us) client_negotiation.cc:769] Sending connection context
0806 09:09:04.263100 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 09:09:04.263794 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":298,"threads_started":1}
I20260806 09:09:04.267256 14925 heartbeater.cc:347] Connected to a master server at 127.27.187.190:35035
I20260806 09:09:04.267644 14925 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:04.268543 14925 heartbeater.cc:511] Master 127.27.187.190:35035 requested a full tablet report, sending...
I20260806 09:09:04.270399 14599 ts_manager.cc:194] Registered new tserver with Master: 61cf626de2ef4fa6b34b14e628d4481f (127.27.187.130:36047)
I20260806 09:09:04.271474 14599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:56619
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:09:04.323062 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:09:04.677572 14790 heartbeater.cc:503] Master 127.27.187.190:35035 was elected leader, sending a full tablet report...
W20260806 09:09:04.696692 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:04.696997 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:04.697211 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:04.704943 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:04.705138 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:04.705327 14933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:09:04.715821 14933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:04.717303 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:04.718806 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:04.733281 14939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:04.733237 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:04.733321 14938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:04.734221 14933 server_base.cc:1034] running on GCE node
I20260806 09:09:04.735009 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:04.736116 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:04.737334 14933 hybrid_clock.cc:648] HybridClock initialized: now 1786007344737285 us; error 40 us; skew 500 ppm
I20260806 09:09:04.737694 14933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:04 dist-test-slave-8tdl krb5kdc[14554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:04.741855 14933 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:09:04.743759 14933 webserver.cc:460] Webserver started at http://127.27.187.131:40585/ using document root <none> and password file <none>
I20260806 09:09:04.744578 14933 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:04.744755 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:04.745121 14933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:04.748490 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "0c7605f63ce241338ea8592e28425ad3"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "4ff69ee95d92457243f3c1e05e7cf955"
server_key_iv: "248f43f3681be45e266893bcda93e279"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.749416 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "0c7605f63ce241338ea8592e28425ad3"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "4ff69ee95d92457243f3c1e05e7cf955"
server_key_iv: "248f43f3681be45e266893bcda93e279"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.755825 14933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:09:04.760362 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.762109 14933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:09:04.762418 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "0c7605f63ce241338ea8592e28425ad3"
format_stamp: "Formatted at 2026-08-06 09:09:04 on dist-test-slave-8tdl"
server_key: "4ff69ee95d92457243f3c1e05e7cf955"
server_key_iv: "248f43f3681be45e266893bcda93e279"
server_key_version: "encryptionkey@0"
I20260806 09:09:04.762668 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:04.786108 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:04.791699 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:04.791985 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:04.793110 14933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:04.794574 14933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:04.794652 14933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.794848 14933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:04.794898 14933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:04.830850 14933 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:41181
I20260806 09:09:04.830929 15060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:41181 every 8 connection(s)
I20260806 09:09:04.832274 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:09:04.837780 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 14933
I20260806 09:09:04.837998 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:09:04.838610 28398 external_mini_cluster.cc:1428] Setting key 65dcb4c377b86f5869d9ebca7456d37f
Aug 06 09:09:04 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:04.857940 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.835230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:44661 (local address 127.27.187.190:35035)
0806 09:09:04.835756 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.835763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.836846 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 09:09:04.839652 (+ 2806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.839657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.839823 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.839948 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:04.842188 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.843161 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.845294 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.845771 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.845891 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.850379 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:04.850385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:04.850388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:04.850434 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:04.854278 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.855515 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.855520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.855522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.855672 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.856108 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.856112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.856114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.856421 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:04.856546 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.857375 (+ 829us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.857565 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
I20260806 09:09:04.859093 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.835830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:35035 (local address 127.27.187.131:44661)
0806 09:09:04.836647 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:04.836686 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:04.839252 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:04.840380 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:04.840394 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:04.841192 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:04.841982 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.842011 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.843350 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.843355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.845085 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.845101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.846027 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.846033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.846267 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.847177 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:04.847210 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:04.850057 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:04.854466 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:04.854478 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:04.854503 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:04.855381 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:04.855790 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:04.855793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:04.855795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:04.855998 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:04.856539 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:04.856545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:04.856944 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 09:09:04.857188 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 09:09:04.857729 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":339,"threads_started":1}
I20260806 09:09:04.860918 15061 heartbeater.cc:347] Connected to a master server at 127.27.187.190:35035
I20260806 09:09:04.861276 15061 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:04.862077 15061 heartbeater.cc:511] Master 127.27.187.190:35035 requested a full tablet report, sending...
I20260806 09:09:04.863850 14599 ts_manager.cc:194] Registered new tserver with Master: 0c7605f63ce241338ea8592e28425ad3 (127.27.187.131:41181)
I20260806 09:09:04.864653 14599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:44661
I20260806 09:09:04.867841 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:09:04.886833 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.872425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.27.187.190:35035)
0806 09:09:04.872599 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.872604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.873133 (+ 529us) server_negotiation.cc:407] Connection header received
0806 09:09:04.873325 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.873329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.873397 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.873484 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:04.874996 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.875794 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.877108 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.877569 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.877724 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.880807 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:04.880812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:04.880816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:04.880854 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:04.883879 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.884854 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.884858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.884861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.884989 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:04.885432 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:04.885436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:04.885438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:04.885769 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:04.885981 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.886338 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.886490 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 09:09:04.891848 14599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:09:04.899287 14599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:09:04.931416 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.916461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:39859 (local address 127.0.0.1:39516)
0806 09:09:04.917103 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:04.917122 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:04.917240 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:04.918812 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:04.918824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:04.918880 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:04.919512 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.919531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.923563 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.923574 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.927700 (+ 4126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.927725 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.930173 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.930182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.930440 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.930505 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 09:09:04.930664 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 09:09:04.930931 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":967,"threads_started":1}
I20260806 09:09:04.934388 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.917198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.27.187.129:39859)
0806 09:09:04.917922 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.917932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.917962 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:09:04.918072 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.918081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.918398 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.920368 (+ 1970us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:04.920757 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.923266 (+ 2509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.928030 (+ 4764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.929862 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.932168 (+ 2306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.933610 (+ 1442us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.933707 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.933926 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":1735,"threads_started":1}
I20260806 09:09:04.939206 14724 tablet_service.cc:1467] Processing CreateTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb (DEFAULT_TABLE table=test-table [id=bd233b1dd9964eeb88e3790d07b74a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:04.940678 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.914969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:41181 (local address 127.0.0.1:53278)
0806 09:09:04.915816 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:04.915862 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:04.916052 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:04.930893 (+ 14841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:04.930901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:04.930950 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:04.931576 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.931599 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.935255 (+ 3656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.935265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.937620 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.937641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.939771 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.939777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.940000 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.940043 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:09:04.940124 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 09:09:04.940287 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":468,"threads_started":1}
I20260806 09:09:04.941123 14724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0509fa928348e38a0c8fe24c2e3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:04.942004 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.915584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.27.187.131:41181)
0806 09:09:04.929888 (+ 14304us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.929901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.929953 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:09:04.930133 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.930146 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.930525 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.932475 (+ 1950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:04.932888 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.935020 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.937852 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.939441 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.941178 (+ 1737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.941375 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.941463 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.941655 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14085,"thread_start_us":13911,"threads_started":1}
I20260806 09:09:04.947088 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.918307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.27.187.130:36047)
0806 09:09:04.938510 (+ 20203us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:04.938517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:04.938553 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:09:04.938653 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:04.938659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:04.939020 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:04.939288 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:04.939958 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.941570 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.944564 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.945685 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.945947 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.946168 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:04.946466 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 09:09:04.946718 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20029,"thread_start_us":20427,"threads_started":1}
I20260806 09:09:04.947091 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:04.918737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36047 (local address 127.0.0.1:58366)
0806 09:09:04.927974 (+ 9237us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:04.927996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:04.928154 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:04.939283 (+ 11129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:04.939289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:04.939325 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:04.939764 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.939777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.941746 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.941752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.944322 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:04.944342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:04.945976 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:04.945984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:04.946223 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:04.946272 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:09:04.946408 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:09:04.946600 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9081,"thread_start_us":8854,"threads_started":1}
I20260806 09:09:04.946916 14996 tablet_service.cc:1467] Processing CreateTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb (DEFAULT_TABLE table=test-table [id=bd233b1dd9964eeb88e3790d07b74a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:04.948886 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0509fa928348e38a0c8fe24c2e3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:04.952010 14860 tablet_service.cc:1467] Processing CreateTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb (DEFAULT_TABLE table=test-table [id=bd233b1dd9964eeb88e3790d07b74a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:04.953950 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0509fa928348e38a0c8fe24c2e3bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:04.959815 15081 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Bootstrap starting.
I20260806 09:09:04.964766 15081 tablet_bootstrap.cc:654] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:04.965852 15082 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Bootstrap starting.
I20260806 09:09:04.967293 15081 log.cc:824] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:04.970288 15082 tablet_bootstrap.cc:654] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:04.971354 15084 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Bootstrap starting.
I20260806 09:09:04.971855 15082 log.cc:824] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:04.971923 15081 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: No bootstrap required, opened a new log
I20260806 09:09:04.972244 15081 ts_tablet_manager.cc:1397] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260806 09:09:04.975436 15082 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: No bootstrap required, opened a new log
I20260806 09:09:04.975654 15084 tablet_bootstrap.cc:654] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:04.975790 15082 ts_tablet_manager.cc:1397] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260806 09:09:04.977147 15084 log.cc:824] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:04.981209 15081 raft_consensus.cc:348] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.981637 15081 raft_consensus.cc:374] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:04.981736 15081 raft_consensus.cc:729] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af249ae4a4b44af892213700a4a17cab, State: Initialized, Role: FOLLOWER
I20260806 09:09:04.982128 15084 tablet_bootstrap.cc:492] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: No bootstrap required, opened a new log
I20260806 09:09:04.982455 15084 ts_tablet_manager.cc:1397] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 09:09:04.982466 15081 consensus_queue.cc:260] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.984530 15081 ts_tablet_manager.cc:1428] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260806 09:09:04.988209 15082 raft_consensus.cc:348] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.988831 15082 raft_consensus.cc:374] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:04.988979 15082 raft_consensus.cc:729] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7605f63ce241338ea8592e28425ad3, State: Initialized, Role: FOLLOWER
I20260806 09:09:04.989746 15082 consensus_queue.cc:260] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.991103 15061 heartbeater.cc:503] Master 127.27.187.190:35035 was elected leader, sending a full tablet report...
I20260806 09:09:04.992151 15082 ts_tablet_manager.cc:1428] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 09:09:04.994345 15084 raft_consensus.cc:348] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.994956 15084 raft_consensus.cc:374] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:04.995081 15084 raft_consensus.cc:729] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61cf626de2ef4fa6b34b14e628d4481f, State: Initialized, Role: FOLLOWER
I20260806 09:09:04.995891 15084 consensus_queue.cc:260] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:04.997259 14925 heartbeater.cc:503] Master 127.27.187.190:35035 was elected leader, sending a full tablet report...
I20260806 09:09:04.998093 15084 ts_tablet_manager.cc:1428] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
W20260806 09:09:05.084102 15062 tablet.cc:2369] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:05.139954 14791 tablet.cc:2369] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:05.241101 14926 tablet.cc:2369] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:09:05.288810 15088 raft_consensus.cc:482] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:09:05.289265 15088 raft_consensus.cc:504] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:05.291347 15088 leader_election.cc:288] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af249ae4a4b44af892213700a4a17cab (127.27.187.129:39859), 61cf626de2ef4fa6b34b14e628d4481f (127.27.187.130:36047)
I20260806 09:09:05.298188 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.291832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:39859 (local address 127.27.187.131:49395)
0806 09:09:05.292079 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:05.292099 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:05.292266 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:05.292920 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:05.292925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:05.292958 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:05.293225 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:05.293236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.294409 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.294416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:05.295985 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:05.296003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.297271 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.297277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:05.297507 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.297535 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:09:05.297659 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 09:09:05.297838 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":1}
I20260806 09:09:05.298179 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.291896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49395 (local address 127.27.187.129:39859)
0806 09:09:05.292285 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.292291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.292317 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:09:05.292433 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.292442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.292544 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.292659 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:05.293422 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.294249 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.296185 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.297088 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.297284 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.297371 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.297690 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.297895 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:09:05.299578 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.292688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:57531 (local address 127.27.187.130:36047)
0806 09:09:05.292930 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.292938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.293076 (+ 138us) server_negotiation.cc:407] Connection header received
0806 09:09:05.293315 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.293325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.293394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.293505 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:05.294078 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.294931 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.296822 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.297730 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.297922 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.298094 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.299122 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.299281 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:09:05.299809 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.292297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:36047 (local address 127.27.187.131:57531)
0806 09:09:05.292949 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:05.292972 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:05.293149 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:05.293521 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:05.293526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:05.293553 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:05.293904 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:05.293913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.295094 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.295100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:05.296629 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:05.296642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.298624 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.298633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:05.298878 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.298941 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:09:05.299058 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:09:05.299227 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":322,"threads_started":1}
I20260806 09:09:05.299564 14745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0509fa928348e38a0c8fe24c2e3bfb" candidate_uuid: "0c7605f63ce241338ea8592e28425ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af249ae4a4b44af892213700a4a17cab" is_pre_election: true
I20260806 09:09:05.300294 14745 raft_consensus.cc:2454] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c7605f63ce241338ea8592e28425ad3 in term 0.
I20260806 09:09:05.301086 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0509fa928348e38a0c8fe24c2e3bfb" candidate_uuid: "0c7605f63ce241338ea8592e28425ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61cf626de2ef4fa6b34b14e628d4481f" is_pre_election: true
I20260806 09:09:05.301218 14949 leader_election.cc:302] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7605f63ce241338ea8592e28425ad3, af249ae4a4b44af892213700a4a17cab; no voters:
I20260806 09:09:05.301676 15088 raft_consensus.cc:2781] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:09:05.301676 14880 raft_consensus.cc:2454] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c7605f63ce241338ea8592e28425ad3 in term 0.
I20260806 09:09:05.301865 15088 raft_consensus.cc:482] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:09:05.301991 15088 raft_consensus.cc:3037] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:05.304728 15088 raft_consensus.cc:504] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:05.305454 15088 leader_election.cc:288] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [CANDIDATE]: Term 1 election: Requested vote from peers af249ae4a4b44af892213700a4a17cab (127.27.187.129:39859), 61cf626de2ef4fa6b34b14e628d4481f (127.27.187.130:36047)
I20260806 09:09:05.306102 14745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0509fa928348e38a0c8fe24c2e3bfb" candidate_uuid: "0c7605f63ce241338ea8592e28425ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af249ae4a4b44af892213700a4a17cab"
I20260806 09:09:05.306116 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0509fa928348e38a0c8fe24c2e3bfb" candidate_uuid: "0c7605f63ce241338ea8592e28425ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61cf626de2ef4fa6b34b14e628d4481f"
I20260806 09:09:05.306370 14880 raft_consensus.cc:3037] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:05.306368 14745 raft_consensus.cc:3037] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:05.308923 14880 raft_consensus.cc:2454] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c7605f63ce241338ea8592e28425ad3 in term 1.
I20260806 09:09:05.309015 14745 raft_consensus.cc:2454] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c7605f63ce241338ea8592e28425ad3 in term 1.
I20260806 09:09:05.309563 14949 leader_election.cc:302] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7605f63ce241338ea8592e28425ad3, 61cf626de2ef4fa6b34b14e628d4481f; no voters:
I20260806 09:09:05.309965 15088 raft_consensus.cc:2781] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:05.310491 15088 raft_consensus.cc:686] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7605f63ce241338ea8592e28425ad3, State: Running, Role: LEADER
I20260806 09:09:05.311129 15088 consensus_queue.cc:237] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } }
I20260806 09:09:05.316931 14598 catalog_manager.cc:5495] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c7605f63ce241338ea8592e28425ad3 (127.27.187.131). New cstate: current_term: 1 leader_uuid: "0c7605f63ce241338ea8592e28425ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7605f63ce241338ea8592e28425ad3" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 } health_report { overall_health: UNKNOWN } } }
I20260806 09:09:05.403306 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.396412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.27.187.131:41181)
0806 09:09:05.396608 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.396613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.397035 (+ 422us) server_negotiation.cc:407] Connection header received
0806 09:09:05.397317 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.397323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.397420 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.397581 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:09:05.398301 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.399134 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.400761 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.401247 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.401358 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.401962 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:09:05.402535 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:09:05.402713 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.403034 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.403090 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 09:09:05.412528 14880 raft_consensus.cc:1261] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 1 FOLLOWER]: Refusing update from remote peer 0c7605f63ce241338ea8592e28425ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:05.412521 14745 raft_consensus.cc:1261] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 1 FOLLOWER]: Refusing update from remote peer 0c7605f63ce241338ea8592e28425ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:05.413650 15088 consensus_queue.cc:1035] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61cf626de2ef4fa6b34b14e628d4481f" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 36047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:05.414019 15091 consensus_queue.cc:1035] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af249ae4a4b44af892213700a4a17cab" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:05.425143 15095 mvcc.cc:205] Tried to move back new op lower bound from 7315486086797529088 to 7315486086399299584. Current Snapshot: MvccSnapshot[applied={T|T < 7315486086797529088}]
I20260806 09:09:05.427794 15097 mvcc.cc:205] Tried to move back new op lower bound from 7315486086797529088 to 7315486086399299584. Current Snapshot: MvccSnapshot[applied={T|T < 7315486086797529088}]
I20260806 09:09:05.446870 14599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:05.447185 14599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:05.451540 14599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fe2a929027c419e9c2aa2a3ff58e501: Sending DeleteTablet for 3 replicas of tablet fe0509fa928348e38a0c8fe24c2e3bfb
I20260806 09:09:05.452886 14996 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:05 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:53278
I20260806 09:09:05.453233 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:05 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:58366
I20260806 09:09:05.453271 14724 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0509fa928348e38a0c8fe24c2e3bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:05 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:39516
I20260806 09:09:05.458225 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14660
I20260806 09:09:05.467334 15107 tablet_replica.cc:330] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: stopping tablet replica
I20260806 09:09:05.467866 15107 raft_consensus.cc:2229] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:05.468475 15107 raft_consensus.cc:2258] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:05.469399 15106 tablet_replica.cc:330] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: stopping tablet replica
I20260806 09:09:05.469914 15106 raft_consensus.cc:2229] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:05.470495 15106 raft_consensus.cc:2258] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:05.471390 15107 ts_tablet_manager.cc:1905] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:05.472844 15106 ts_tablet_manager.cc:1905] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:05.479310 15107 ts_tablet_manager.cc:1918] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:05.479537 15107 log.cc:1196] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/fe0509fa928348e38a0c8fe24c2e3bfb
I20260806 09:09:05.480206 15106 ts_tablet_manager.cc:1918] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:05.480283 15107 ts_tablet_manager.cc:1939] T fe0509fa928348e38a0c8fe24c2e3bfb P af249ae4a4b44af892213700a4a17cab: Deleting consensus metadata
I20260806 09:09:05.480406 15106 log.cc:1196] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/fe0509fa928348e38a0c8fe24c2e3bfb
I20260806 09:09:05.480985 15106 ts_tablet_manager.cc:1939] T fe0509fa928348e38a0c8fe24c2e3bfb P 61cf626de2ef4fa6b34b14e628d4481f: Deleting consensus metadata
I20260806 09:09:05.482565 14583 catalog_manager.cc:4841] TS af249ae4a4b44af892213700a4a17cab (127.27.187.129:39859): tablet fe0509fa928348e38a0c8fe24c2e3bfb (table test-table [id=bd233b1dd9964eeb88e3790d07b74a85]) successfully deleted
I20260806 09:09:05.483026 14583 catalog_manager.cc:4841] TS 61cf626de2ef4fa6b34b14e628d4481f (127.27.187.130:36047): tablet fe0509fa928348e38a0c8fe24c2e3bfb (table test-table [id=bd233b1dd9964eeb88e3790d07b74a85]) successfully deleted
I20260806 09:09:05.484134 15105 tablet_replica.cc:330] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: stopping tablet replica
I20260806 09:09:05.484632 15105 raft_consensus.cc:2229] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:09:05.485224 15105 raft_consensus.cc:2258] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:05.487228 15105 ts_tablet_manager.cc:1905] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:09:05 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:09:05.495365 15105 ts_tablet_manager.cc:1918] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:05.495612 15105 log.cc:1196] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/fe0509fa928348e38a0c8fe24c2e3bfb
I20260806 09:09:05.496153 15105 ts_tablet_manager.cc:1939] T fe0509fa928348e38a0c8fe24c2e3bfb P 0c7605f63ce241338ea8592e28425ad3: Deleting consensus metadata
I20260806 09:09:05.497956 14583 catalog_manager.cc:4841] TS 0c7605f63ce241338ea8592e28425ad3 (127.27.187.131:41181): tablet fe0509fa928348e38a0c8fe24c2e3bfb (table test-table [id=bd233b1dd9964eeb88e3790d07b74a85]) successfully deleted
I20260806 09:09:05.501657 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.460476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.27.187.129:39859)
0806 09:09:05.460738 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.460748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.480001 (+ 19253us) server_negotiation.cc:407] Connection header received
0806 09:09:05.480339 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.480348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.480448 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.480541 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:05.482703 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.483896 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.485939 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.486627 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.486769 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.493285 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:05.493309 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:05.493332 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:05.493397 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:05.497733 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.498873 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.498885 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.498889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.499084 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.499609 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.499617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.499622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.500179 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:05.500545 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.500972 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.501242 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:09:05.502460 14785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:05.599586 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14660
I20260806 09:09:05.625247 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14797
Aug 06 09:09:05 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:09:05.642732 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.626216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.27.187.130:36047)
0806 09:09:05.626417 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.626423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.626477 (+ 54us) server_negotiation.cc:407] Connection header received
0806 09:09:05.626763 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.626794 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.626886 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.627010 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:05.628497 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.629230 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.630626 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.631129 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.631279 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.636055 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:05.636074 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:05.636092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:05.636139 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:05.639454 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.640320 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.640329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.640331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.640474 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.640931 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.640938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.640941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.641369 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:05.641702 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.642045 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.642300 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:09:05.643544 14920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:05.755125 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14797
I20260806 09:09:05.782929 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14933
Aug 06 09:09:05 dist-test-slave-8tdl krb5kdc[14554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:09:05.802119 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:05.783995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.27.187.131:41181)
0806 09:09:05.784577 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:05.784582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:05.784599 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:09:05.784661 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:05.784665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:05.784739 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:05.784876 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:05.786319 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.787367 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.788808 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:05.789310 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:05.789480 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:05.795015 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:05.795031 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:05.795062 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:05.795105 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:05.798855 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.799690 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.799704 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.799706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.799860 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:05.800307 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:05.800311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:05.800313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:05.800744 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:05.801126 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:05.801439 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:09:05.801700 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260806 09:09:05.802925 15056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:05.932652 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14933
I20260806 09:09:05.955286 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 14567
I20260806 09:09:05.956602 14629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:06.075338 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 14567
2026-08-06T09:09:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5181 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T09:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:06Z Disabled control of system clock
I20260806 09:09:06.176271 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45015
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:45015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:09:06.553396 15116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:06.553951 15116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:06.554107 15116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:06.562232 15116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:06.562377 15116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:06.562419 15116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:06.562480 15116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:06.562526 15116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:06.572736 15116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:45015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:45015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:06.574118 15116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:06.576025 15116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:06.590565 15121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:06.590507 15122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:06.590703 15116 server_base.cc:1034] running on GCE node
W20260806 09:09:06.590487 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:06.592068 15116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:06.593848 15116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:06.595381 15116 hybrid_clock.cc:648] HybridClock initialized: now 1786007346595314 us; error 49 us; skew 500 ppm
I20260806 09:09:06.595886 15116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:06.598732 15116 webserver.cc:460] Webserver started at http://127.27.187.190:36949/ using document root <none> and password file <none>
I20260806 09:09:06.599572 15116 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:06.599682 15116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:06.600117 15116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:06.603477 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "a9b3804a0947442597d420c11b915ce7"
format_stamp: "Formatted at 2026-08-06 09:09:06 on dist-test-slave-8tdl"
server_key: "6af2380779c0251a4a8bc1c30f4e0fa7"
server_key_iv: "e7ef16ac3b380d4e7b6a823920e33bb1"
server_key_version: "encryptionkey@0"
I20260806 09:09:06.604445 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "a9b3804a0947442597d420c11b915ce7"
format_stamp: "Formatted at 2026-08-06 09:09:06 on dist-test-slave-8tdl"
server_key: "6af2380779c0251a4a8bc1c30f4e0fa7"
server_key_iv: "e7ef16ac3b380d4e7b6a823920e33bb1"
server_key_version: "encryptionkey@0"
I20260806 09:09:06.610870 15116 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 09:09:06.615269 15130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:06.616864 15116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:09:06.617089 15116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "a9b3804a0947442597d420c11b915ce7"
format_stamp: "Formatted at 2026-08-06 09:09:06 on dist-test-slave-8tdl"
server_key: "6af2380779c0251a4a8bc1c30f4e0fa7"
server_key_iv: "e7ef16ac3b380d4e7b6a823920e33bb1"
server_key_version: "encryptionkey@0"
I20260806 09:09:06.617431 15116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:06.644359 15116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:06.645440 15116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:06.645809 15116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:06.669750 15116 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:45015
I20260806 09:09:06.669780 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:45015 every 8 connection(s)
I20260806 09:09:06.671470 15116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:06.672039 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15116
I20260806 09:09:06.672345 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:06.672952 28398 external_mini_cluster.cc:1428] Setting key 40d8122d53ea0f3060a1ebe92564258d
I20260806 09:09:06.676548 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:06.688835 15182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: Bootstrap starting.
I20260806 09:09:06.689039 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:06.675121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.27.187.190:45015)
0806 09:09:06.676269 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:06.676282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:06.676342 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:09:06.678384 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:06.678415 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:06.679079 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:06.679621 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:06.680316 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:06.681833 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:06.683583 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:06.684411 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:06.684654 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:06.686195 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:06.686211 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:06.686230 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:06.686997 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:06.687092 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:06.687118 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:06.687266 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:06.687537 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:09:06.687905 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":438,"threads_started":1}
I20260806 09:09:06.692925 15182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:06.694003 15182 log.cc:824] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:06.697571 15182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: No bootstrap required, opened a new log
I20260806 09:09:06.705925 15182 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } }
I20260806 09:09:06.706414 15182 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:06.706525 15182 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b3804a0947442597d420c11b915ce7, State: Initialized, Role: FOLLOWER
I20260806 09:09:06.707208 15182 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } }
I20260806 09:09:06.707458 15182 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:06.707597 15182 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:06.707773 15182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:06.710103 15182 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } }
I20260806 09:09:06.710816 15182 leader_election.cc:302] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9b3804a0947442597d420c11b915ce7; no voters:
I20260806 09:09:06.711542 15182 leader_election.cc:288] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:06.711722 15187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:06.713158 15187 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [term 1 LEADER]: Becoming Leader. State: Replica: a9b3804a0947442597d420c11b915ce7, State: Running, Role: LEADER
I20260806 09:09:06.713896 15187 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } }
I20260806 09:09:06.714072 15182 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:06.717141 15188 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9b3804a0947442597d420c11b915ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } } }
I20260806 09:09:06.717669 15188 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:06.717180 15189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9b3804a0947442597d420c11b915ce7. Latest consensus state: current_term: 1 leader_uuid: "a9b3804a0947442597d420c11b915ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b3804a0947442597d420c11b915ce7" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 45015 } } }
I20260806 09:09:06.717787 15189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:06.720098 15194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:06.725139 15194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:06.734911 15194 catalog_manager.cc:1347] Generated new cluster ID: 9092e405f7ad4033a46f26ffce2f8cf0
I20260806 09:09:06.735059 15194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:06.762236 15194 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:06.764341 15194 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:06.777320 15194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: Generated new TSK 0
I20260806 09:09:06.778839 15194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:06.784740 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--builtin_ntp_servers=127.27.187.148:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:09:07.175594 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:07.175940 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:07.176033 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:07.185240 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:07.185369 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:07.185483 15207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:09:07.195930 15207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:07.197310 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:07.199303 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:07.214385 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:07.214385 15213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:07.214365 15215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:07.215483 15207 server_base.cc:1034] running on GCE node
I20260806 09:09:07.216259 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:07.217428 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:07.218796 15207 hybrid_clock.cc:648] HybridClock initialized: now 1786007347218702 us; error 74 us; skew 500 ppm
I20260806 09:09:07.219231 15207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:07.222532 15207 webserver.cc:460] Webserver started at http://127.27.187.129:32865/ using document root <none> and password file <none>
I20260806 09:09:07.223387 15207 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:07.223490 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:07.223883 15207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:07.227459 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "4356e24703b44e26ad260c5269241158"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "f6bbbe8b4fb64610e9b9d7cbd9f57825"
server_key_iv: "8f7724d60da1e5bf21d39fc9a795d8c1"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.228477 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "4356e24703b44e26ad260c5269241158"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "f6bbbe8b4fb64610e9b9d7cbd9f57825"
server_key_iv: "8f7724d60da1e5bf21d39fc9a795d8c1"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.235039 15207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:09:07.239471 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.241176 15207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:09:07.241392 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "4356e24703b44e26ad260c5269241158"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "f6bbbe8b4fb64610e9b9d7cbd9f57825"
server_key_iv: "8f7724d60da1e5bf21d39fc9a795d8c1"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.241663 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:07.263886 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:07.264883 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:07.265502 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:07.266865 15207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:07.268746 15207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:07.268885 15207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.269035 15207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:07.269129 15207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.306114 15207 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43007
I20260806 09:09:07.306221 15333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43007 every 8 connection(s)
I20260806 09:09:07.307585 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:09:07.312175 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15207
I20260806 09:09:07.312377 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:09:07.312963 28398 external_mini_cluster.cc:1428] Setting key dc9194a1659c6c3ac393fde1f3df520f
I20260806 09:09:07.316896 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--builtin_ntp_servers=127.27.187.148:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:09:07.325974 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:07.310660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:36687 (local address 127.27.187.190:45015)
0806 09:09:07.311318 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:07.311325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:07.312956 (+ 1631us) server_negotiation.cc:407] Connection header received
0806 09:09:07.315484 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:07.315490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:07.315579 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:07.315726 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:07.317853 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.318968 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.322115 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.322808 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.322960 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:07.324880 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:07.324889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:07.324896 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:07.324925 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:07.324977 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:07.324994 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:07.325090 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:07.325410 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:09:07.325529 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
I20260806 09:09:07.327513 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:07.311348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45015 (local address 127.27.187.129:36687)
0806 09:09:07.312725 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:07.312780 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:07.315133 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:07.316025 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:07.316039 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:07.316215 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:07.317503 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:07.317541 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.319350 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.319359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:07.321813 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:07.321837 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.323099 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.323111 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:07.323490 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:07.324424 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:07.324430 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:07.324504 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:07.324636 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:07.325136 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:07.325151 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:07.325162 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:09:07.325389 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 09:09:07.325918 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":396,"threads_started":1}
I20260806 09:09:07.329595 15334 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45015
I20260806 09:09:07.329998 15334 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:07.331029 15334 heartbeater.cc:511] Master 127.27.187.190:45015 requested a full tablet report, sending...
I20260806 09:09:07.333631 15147 ts_manager.cc:194] Registered new tserver with Master: 4356e24703b44e26ad260c5269241158 (127.27.187.129:43007)
I20260806 09:09:07.335381 15147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.129:36687
W20260806 09:09:07.697935 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:07.698311 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:07.698462 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:07.706383 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:07.706511 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:07.706673 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:09:07.716748 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:07.718411 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:07.721100 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:07.739981 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:07.740013 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:07.739974 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:07.740840 15338 server_base.cc:1034] running on GCE node
I20260806 09:09:07.741530 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:07.742784 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:07.744109 15338 hybrid_clock.cc:648] HybridClock initialized: now 1786007347744040 us; error 59 us; skew 500 ppm
I20260806 09:09:07.744521 15338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:07.747330 15338 webserver.cc:460] Webserver started at http://127.27.187.130:33139/ using document root <none> and password file <none>
I20260806 09:09:07.748132 15338 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:07.748283 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:07.748628 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:07.751982 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "1043b5450eb8a91b8811f3ab3746a126"
server_key_iv: "def6966cecf05cd02125fa6f5edf3bc6"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.752960 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "1043b5450eb8a91b8811f3ab3746a126"
server_key_iv: "def6966cecf05cd02125fa6f5edf3bc6"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.759370 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:09:07.763676 15352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.765465 15338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:09:07.765658 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97"
format_stamp: "Formatted at 2026-08-06 09:09:07 on dist-test-slave-8tdl"
server_key: "1043b5450eb8a91b8811f3ab3746a126"
server_key_iv: "def6966cecf05cd02125fa6f5edf3bc6"
server_key_version: "encryptionkey@0"
I20260806 09:09:07.765933 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:07.785300 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:07.786329 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:07.786749 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:07.787919 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:07.789625 15338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:07.789728 15338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.789883 15338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:07.789989 15338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:07.829811 15338 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:33707
I20260806 09:09:07.829918 15465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:33707 every 8 connection(s)
I20260806 09:09:07.831369 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:09:07.836524 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15338
I20260806 09:09:07.836717 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:09:07.837213 28398 external_mini_cluster.cc:1428] Setting key 3a699f6f24928331a23bd9811d6c8b0c
I20260806 09:09:07.840427 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--builtin_ntp_servers=127.27.187.148:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:09:07.850443 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:07.834362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:40907 (local address 127.27.187.190:45015)
0806 09:09:07.834890 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:07.834898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:07.836319 (+ 1421us) server_negotiation.cc:407] Connection header received
0806 09:09:07.838488 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:07.838494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:07.838597 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:07.838740 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:07.840111 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.840991 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.842703 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.843141 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.843302 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:07.849341 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:07.849347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:07.849350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:07.849375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:07.849421 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:07.849435 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:07.849556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:07.849845 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 09:09:07.849980 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":235,"threads_started":1}
I20260806 09:09:07.851665 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:07.835070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45015 (local address 127.27.187.130:40907)
0806 09:09:07.836101 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:07.836140 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:07.838189 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:07.839024 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:07.839035 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:07.839180 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:07.839924 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:07.839944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.841170 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.841175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:07.842499 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:07.842514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:07.843412 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:07.843421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:07.843685 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:07.848847 (+ 5162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:07.848851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:07.848908 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:07.849020 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:07.849559 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:07.849577 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:07.849588 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:09:07.849809 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:09:07.850348 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":302,"threads_started":1}
I20260806 09:09:07.853523 15466 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45015
I20260806 09:09:07.853933 15466 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:07.855077 15466 heartbeater.cc:511] Master 127.27.187.190:45015 requested a full tablet report, sending...
I20260806 09:09:07.857013 15147 ts_manager.cc:194] Registered new tserver with Master: 1e3ed735c0134cf1b8c65ac1c40a4d97 (127.27.187.130:33707)
I20260806 09:09:07.858358 15147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.130:40907
W20260806 09:09:08.212371 15470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:08.212692 15470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:08.212790 15470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:08.220979 15470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:08.221109 15470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:08.221230 15470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:09:08.231340 15470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:45015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:08.232786 15470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:08.234824 15470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:08.250382 15476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:08.250353 15478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:08.250411 15475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:08.251541 15470 server_base.cc:1034] running on GCE node
I20260806 09:09:08.252293 15470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:08.253412 15470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:08.254765 15470 hybrid_clock.cc:648] HybridClock initialized: now 1786007348254638 us; error 36 us; skew 500 ppm
I20260806 09:09:08.255208 15470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:08.257984 15470 webserver.cc:460] Webserver started at http://127.27.187.131:40237/ using document root <none> and password file <none>
I20260806 09:09:08.258843 15470 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:08.258973 15470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:08.259332 15470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:08.262431 15470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12"
format_stamp: "Formatted at 2026-08-06 09:09:08 on dist-test-slave-8tdl"
server_key: "b102c894e31c3c74deef0ed39a2aff5e"
server_key_iv: "bc551a210f4d305ce98c06f76029228e"
server_key_version: "encryptionkey@0"
I20260806 09:09:08.263401 15470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12"
format_stamp: "Formatted at 2026-08-06 09:09:08 on dist-test-slave-8tdl"
server_key: "b102c894e31c3c74deef0ed39a2aff5e"
server_key_iv: "bc551a210f4d305ce98c06f76029228e"
server_key_version: "encryptionkey@0"
I20260806 09:09:08.269085 15470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:09:08.273263 15484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:08.274863 15470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:08.275071 15470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12"
format_stamp: "Formatted at 2026-08-06 09:09:08 on dist-test-slave-8tdl"
server_key: "b102c894e31c3c74deef0ed39a2aff5e"
server_key_iv: "bc551a210f4d305ce98c06f76029228e"
server_key_version: "encryptionkey@0"
I20260806 09:09:08.275321 15470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:08.293183 15470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:08.294138 15470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:08.294498 15470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:08.295691 15470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:08.297560 15470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:08.297669 15470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:08.297785 15470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:08.297865 15470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:08.335049 15470 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:46511
I20260806 09:09:08.335114 15597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:46511 every 8 connection(s)
I20260806 09:09:08.336530 15470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:09:08.339141 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15470
I20260806 09:09:08.339383 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:09:08.339838 15334 heartbeater.cc:503] Master 127.27.187.190:45015 was elected leader, sending a full tablet report...
I20260806 09:09:08.340031 28398 external_mini_cluster.cc:1428] Setting key 9b28e2bec936165ef4c524f9b000d574
I20260806 09:09:08.351485 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.339511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:32969 (local address 127.27.187.190:45015)
0806 09:09:08.339764 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.339771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.341827 (+ 2056us) server_negotiation.cc:407] Connection header received
0806 09:09:08.344230 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.344235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.344360 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.344453 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:08.345722 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.346734 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.348519 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.348960 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.349071 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.350636 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:08.350642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:08.350645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:08.350668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:08.350791 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:08.350808 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:08.350891 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.351128 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.351223 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:09:08.352697 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.340290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:45015 (local address 127.27.187.131:32969)
0806 09:09:08.341628 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.341692 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.343942 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.344691 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.344703 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.344806 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:08.345549 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.345569 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.346898 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.346903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.348356 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.348371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.349259 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.349266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.349489 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.350292 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:08.350295 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:08.350338 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:08.350426 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:08.350912 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:08.350921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:08.350928 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.351093 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.351499 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":408,"threads_started":1}
I20260806 09:09:08.354526 15598 heartbeater.cc:347] Connected to a master server at 127.27.187.190:45015
I20260806 09:09:08.354941 15598 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:08.355803 15598 heartbeater.cc:511] Master 127.27.187.190:45015 requested a full tablet report, sending...
I20260806 09:09:08.357331 15147 ts_manager.cc:194] Registered new tserver with Master: 1fa0214f7ac24e4fbdb0e74fcfed4d12 (127.27.187.131:46511)
I20260806 09:09:08.358256 15147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.131:32969
I20260806 09:09:08.369381 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:09:08.381915 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.375183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.27.187.190:45015)
0806 09:09:08.375461 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.375470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.375782 (+ 312us) server_negotiation.cc:407] Connection header received
0806 09:09:08.376007 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.376013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.376165 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.376289 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:08.376847 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.377621 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.379138 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.379586 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.379734 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.381079 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:08.381083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:08.381093 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:08.381113 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:08.381148 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:08.381161 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:08.381327 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.381525 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.381587 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:09:08.386940 15147 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:09:08.394893 15147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:09:08.420737 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.409143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:33707 (local address 127.0.0.1:50668)
0806 09:09:08.411349 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.411401 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.411747 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.413151 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.413163 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.413211 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:08.413744 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.413765 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.415670 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.415677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.417192 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.417206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.419635 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.419644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.419912 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.419971 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.420120 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.420365 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1847,"thread_start_us":3634,"threads_started":1}
I20260806 09:09:08.420866 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.409240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.27.187.130:33707)
0806 09:09:08.410159 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.410170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.411534 (+ 1364us) server_negotiation.cc:407] Connection header received
0806 09:09:08.412118 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.412132 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.412568 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.412773 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:08.413951 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.415529 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.417397 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.419061 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.419246 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.419488 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.420237 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.420465 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":616,"threads_started":1}
I20260806 09:09:08.423405 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.409885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:46511 (local address 127.0.0.1:43802)
0806 09:09:08.413103 (+ 3218us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.413122 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.413263 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.414951 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.414957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.414992 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:08.415476 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.415494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.417638 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.417646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.420313 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.420332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.422339 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.422348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.422539 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.422574 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.422744 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.422901 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3064,"mutex_wait_us":36,"spinlock_wait_cycles":21120,"thread_start_us":2826,"threads_started":1}
I20260806 09:09:08.423344 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.410549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.27.187.131:46511)
0806 09:09:08.413975 (+ 3426us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.413986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.414024 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:09:08.414152 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.414163 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.414413 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.414611 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:08.415806 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.417480 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.420595 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.422113 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.422391 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.422662 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.422781 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.422952 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3170,"thread_start_us":3003,"threads_started":1}
I20260806 09:09:08.425433 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.409849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48188)
0806 09:09:08.413009 (+ 3160us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.413031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.413288 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.414428 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.414435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.414468 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:08.415076 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.415095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.417415 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.417424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.420341 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.420357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.424549 (+ 4192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.424556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.424750 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.424790 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.424888 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.425056 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2993,"mutex_wait_us":191,"thread_start_us":893,"threads_started":1}
I20260806 09:09:08.425640 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.410636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.27.187.129:43007)
0806 09:09:08.412581 (+ 1945us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.412594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.413155 (+ 561us) server_negotiation.cc:407] Connection header received
0806 09:09:08.413525 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.413536 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.413888 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.414086 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:08.415331 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.417179 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.420623 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.422426 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.424169 (+ 1743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.424413 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.425023 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.425227 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1759,"spinlock_wait_cycles":25984,"thread_start_us":1587,"threads_started":1}
I20260806 09:09:08.426915 15400 tablet_service.cc:1467] Processing CreateTablet for tablet 865adfc7acff445491ae7e931ea4263c (DEFAULT_TABLE table=test-table [id=6e06fc357c994674b8c980640404f6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:08.428318 15400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865adfc7acff445491ae7e931ea4263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:08.427819 15532 tablet_service.cc:1467] Processing CreateTablet for tablet 865adfc7acff445491ae7e931ea4263c (DEFAULT_TABLE table=test-table [id=6e06fc357c994674b8c980640404f6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:08.429342 15532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865adfc7acff445491ae7e931ea4263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:08.429524 15269 tablet_service.cc:1467] Processing CreateTablet for tablet 865adfc7acff445491ae7e931ea4263c (DEFAULT_TABLE table=test-table [id=6e06fc357c994674b8c980640404f6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:08.431085 15269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865adfc7acff445491ae7e931ea4263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:08.441390 15616 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Bootstrap starting.
I20260806 09:09:08.444813 15617 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Bootstrap starting.
I20260806 09:09:08.446509 15618 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Bootstrap starting.
I20260806 09:09:08.447597 15616 tablet_bootstrap.cc:654] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:08.449170 15616 log.cc:824] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:08.449725 15618 tablet_bootstrap.cc:654] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:08.449852 15617 tablet_bootstrap.cc:654] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:08.451252 15618 log.cc:824] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:08.451206 15617 log.cc:824] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:08.454269 15616 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: No bootstrap required, opened a new log
I20260806 09:09:08.454617 15618 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: No bootstrap required, opened a new log
I20260806 09:09:08.454716 15616 ts_tablet_manager.cc:1397] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260806 09:09:08.454969 15618 ts_tablet_manager.cc:1397] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:09:08.461793 15617 tablet_bootstrap.cc:492] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: No bootstrap required, opened a new log
I20260806 09:09:08.462162 15617 ts_tablet_manager.cc:1397] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260806 09:09:08.463374 15618 raft_consensus.cc:348] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.463943 15618 raft_consensus.cc:374] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:08.464104 15618 raft_consensus.cc:729] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4356e24703b44e26ad260c5269241158, State: Initialized, Role: FOLLOWER
I20260806 09:09:08.464838 15618 consensus_queue.cc:260] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.466974 15618 ts_tablet_manager.cc:1428] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260806 09:09:08.467418 15616 raft_consensus.cc:348] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.468065 15616 raft_consensus.cc:374] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:08.468189 15616 raft_consensus.cc:729] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3ed735c0134cf1b8c65ac1c40a4d97, State: Initialized, Role: FOLLOWER
I20260806 09:09:08.468899 15616 consensus_queue.cc:260] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.470323 15466 heartbeater.cc:503] Master 127.27.187.190:45015 was elected leader, sending a full tablet report...
I20260806 09:09:08.471297 15616 ts_tablet_manager.cc:1428] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260806 09:09:08.475430 15617 raft_consensus.cc:348] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.475942 15617 raft_consensus.cc:374] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:08.476089 15617 raft_consensus.cc:729] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa0214f7ac24e4fbdb0e74fcfed4d12, State: Initialized, Role: FOLLOWER
I20260806 09:09:08.476763 15617 consensus_queue.cc:260] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.477975 15598 heartbeater.cc:503] Master 127.27.187.190:45015 was elected leader, sending a full tablet report...
I20260806 09:09:08.478929 15617 ts_tablet_manager.cc:1428] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260806 09:09:08.560768 15335 tablet.cc:2369] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:08.583835 15467 tablet.cc:2369] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:08.588343 15599 tablet.cc:2369] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:09:08.612414 15622 raft_consensus.cc:482] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:09:08.612782 15622 raft_consensus.cc:504] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.614879 15622 leader_election.cc:288] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e3ed735c0134cf1b8c65ac1c40a4d97 (127.27.187.130:33707), 1fa0214f7ac24e4fbdb0e74fcfed4d12 (127.27.187.131:46511)
I20260806 09:09:08.623536 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.615439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:39363 (local address 127.27.187.130:33707)
0806 09:09:08.615720 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.615731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.616605 (+ 874us) server_negotiation.cc:407] Connection header received
0806 09:09:08.616773 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.616779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.616889 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.616993 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:08.617735 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.618594 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.621289 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.622346 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.622537 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.622614 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.623053 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.623208 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:09:08.623700 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.615288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:33707 (local address 127.27.187.129:39363)
0806 09:09:08.616471 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.616489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.616639 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.617026 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.617031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.617059 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:08.617555 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.617566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.618807 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.618813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.621027 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.621044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.622577 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.622585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.622824 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.622880 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.622998 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.623166 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":95,"spinlock_wait_cycles":41728,"thread_start_us":762,"threads_started":1}
I20260806 09:09:08.623692 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.615516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:54861 (local address 127.27.187.131:46511)
0806 09:09:08.615802 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.615813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.616782 (+ 969us) server_negotiation.cc:407] Connection header received
0806 09:09:08.616979 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.616984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.617072 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.617191 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:08.617927 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.618831 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.621386 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.622568 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.622804 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.622881 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.623261 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.623413 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:09:08.623802 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.615294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:46511 (local address 127.27.187.129:54861)
0806 09:09:08.616635 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.616657 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:08.616809 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:08.617228 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:08.617232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:08.617265 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:08.617726 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.617739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.619018 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.619124 (+ 106us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.621054 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:08.621069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.622841 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.622847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:08.623061 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.623098 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:09:08.623217 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:09:08.623377 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":11,"thread_start_us":811,"threads_started":1}
I20260806 09:09:08.625079 15552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865adfc7acff445491ae7e931ea4263c" candidate_uuid: "4356e24703b44e26ad260c5269241158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" is_pre_election: true
I20260806 09:09:08.625082 15420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865adfc7acff445491ae7e931ea4263c" candidate_uuid: "4356e24703b44e26ad260c5269241158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" is_pre_election: true
I20260806 09:09:08.625670 15420 raft_consensus.cc:2454] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356e24703b44e26ad260c5269241158 in term 0.
I20260806 09:09:08.625672 15552 raft_consensus.cc:2454] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4356e24703b44e26ad260c5269241158 in term 0.
I20260806 09:09:08.626487 15225 leader_election.cc:302] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e3ed735c0134cf1b8c65ac1c40a4d97, 1fa0214f7ac24e4fbdb0e74fcfed4d12, 4356e24703b44e26ad260c5269241158; no voters:
I20260806 09:09:08.627048 15622 raft_consensus.cc:2781] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:09:08.627281 15622 raft_consensus.cc:482] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:09:08.627426 15622 raft_consensus.cc:3037] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:08.629932 15622 raft_consensus.cc:504] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.630859 15622 leader_election.cc:288] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [CANDIDATE]: Term 1 election: Requested vote from peers 1e3ed735c0134cf1b8c65ac1c40a4d97 (127.27.187.130:33707), 1fa0214f7ac24e4fbdb0e74fcfed4d12 (127.27.187.131:46511)
I20260806 09:09:08.631443 15420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865adfc7acff445491ae7e931ea4263c" candidate_uuid: "4356e24703b44e26ad260c5269241158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97"
I20260806 09:09:08.631512 15552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865adfc7acff445491ae7e931ea4263c" candidate_uuid: "4356e24703b44e26ad260c5269241158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12"
I20260806 09:09:08.631682 15420 raft_consensus.cc:3037] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:08.631810 15552 raft_consensus.cc:3037] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:08.634213 15420 raft_consensus.cc:2454] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356e24703b44e26ad260c5269241158 in term 1.
I20260806 09:09:08.634372 15552 raft_consensus.cc:2454] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4356e24703b44e26ad260c5269241158 in term 1.
I20260806 09:09:08.634832 15222 leader_election.cc:302] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3ed735c0134cf1b8c65ac1c40a4d97, 4356e24703b44e26ad260c5269241158; no voters:
I20260806 09:09:08.635262 15622 raft_consensus.cc:2781] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:08.635814 15622 raft_consensus.cc:686] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 1 LEADER]: Becoming Leader. State: Replica: 4356e24703b44e26ad260c5269241158, State: Running, Role: LEADER
I20260806 09:09:08.636355 15622 consensus_queue.cc:237] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } }
I20260806 09:09:08.642241 15146 catalog_manager.cc:5495] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4356e24703b44e26ad260c5269241158 (127.27.187.129). New cstate: current_term: 1 leader_uuid: "4356e24703b44e26ad260c5269241158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4356e24703b44e26ad260c5269241158" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 } health_report { overall_health: UNKNOWN } } }
I20260806 09:09:08.671752 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.664134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.27.187.129:43007)
0806 09:09:08.664364 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.664373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.664715 (+ 342us) server_negotiation.cc:407] Connection header received
0806 09:09:08.665029 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.665035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.665266 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.665454 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:09:08.665937 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.666892 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.668660 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.669355 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.669515 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.670216 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:09:08.670977 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:09:08.671164 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.671386 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.671439 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:09:08.682550 15420 raft_consensus.cc:1261] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 1 FOLLOWER]: Refusing update from remote peer 4356e24703b44e26ad260c5269241158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:08.682585 15552 raft_consensus.cc:1261] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 1 FOLLOWER]: Refusing update from remote peer 4356e24703b44e26ad260c5269241158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:08.683847 15622 consensus_queue.cc:1035] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa0214f7ac24e4fbdb0e74fcfed4d12" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:08.684230 15627 consensus_queue.cc:1035] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3ed735c0134cf1b8c65ac1c40a4d97" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 33707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:08.702347 15631 mvcc.cc:205] Tried to move back new op lower bound from 7315486100190724096 to 7315486100018749440. Current Snapshot: MvccSnapshot[applied={T|T < 7315486100190724096}]
I20260806 09:09:08.710170 15632 mvcc.cc:205] Tried to move back new op lower bound from 7315486100190724096 to 7315486100018749440. Current Snapshot: MvccSnapshot[applied={T|T < 7315486100190724096}]
I20260806 09:09:08.727226 15147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:08.727571 15147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:08.731967 15147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9b3804a0947442597d420c11b915ce7: Sending DeleteTablet for 3 replicas of tablet 865adfc7acff445491ae7e931ea4263c
I20260806 09:09:08.733615 15400 tablet_service.cc:1514] Processing DeleteTablet for tablet 865adfc7acff445491ae7e931ea4263c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:08 UTC) from {username='slave'} at 127.0.0.1:50668
I20260806 09:09:08.733747 15532 tablet_service.cc:1514] Processing DeleteTablet for tablet 865adfc7acff445491ae7e931ea4263c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:08 UTC) from {username='slave'} at 127.0.0.1:43802
I20260806 09:09:08.734216 15269 tablet_service.cc:1514] Processing DeleteTablet for tablet 865adfc7acff445491ae7e931ea4263c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:08 UTC) from {username='slave'} at 127.0.0.1:48188
I20260806 09:09:08.738248 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15207
I20260806 09:09:08.745573 15642 tablet_replica.cc:330] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: stopping tablet replica
I20260806 09:09:08.746312 15642 raft_consensus.cc:2229] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:09:08.747309 15642 raft_consensus.cc:2258] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:08.749018 15640 tablet_replica.cc:330] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: stopping tablet replica
I20260806 09:09:08.749655 15640 raft_consensus.cc:2229] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:08.750344 15640 raft_consensus.cc:2258] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:08.751281 15641 tablet_replica.cc:330] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: stopping tablet replica
I20260806 09:09:08.751749 15641 raft_consensus.cc:2229] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:08.752365 15641 raft_consensus.cc:2258] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:08.753702 15642 ts_tablet_manager.cc:1905] T 865adfc7acff445491ae7e931ea4263c P 4356e24703b44e26ad260c5269241158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:08.754896 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.741175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.27.187.129:43007)
0806 09:09:08.741387 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.741394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.742400 (+ 1006us) server_negotiation.cc:407] Connection header received
0806 09:09:08.742763 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.742769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.742902 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.743018 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:08.745974 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.747642 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.750013 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.750817 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.750953 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.752827 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:08.752844 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:08.752865 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:08.753708 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:08.753805 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:08.753835 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:08.753942 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.754281 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.754422 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:09:08.754907 15640 ts_tablet_manager.cc:1905] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:08.756448 15329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:08.760654 15641 ts_tablet_manager.cc:1905] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:08.763324 15640 ts_tablet_manager.cc:1918] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:08.763523 15640 log.cc:1196] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/865adfc7acff445491ae7e931ea4263c
I20260806 09:09:08.764185 15640 ts_tablet_manager.cc:1939] T 865adfc7acff445491ae7e931ea4263c P 1e3ed735c0134cf1b8c65ac1c40a4d97: Deleting consensus metadata
I20260806 09:09:08.766412 15131 catalog_manager.cc:4841] TS 1e3ed735c0134cf1b8c65ac1c40a4d97 (127.27.187.130:33707): tablet 865adfc7acff445491ae7e931ea4263c (table test-table [id=6e06fc357c994674b8c980640404f6ec]) successfully deleted
I20260806 09:09:08.768025 15641 ts_tablet_manager.cc:1918] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:08.768213 15641 log.cc:1196] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/865adfc7acff445491ae7e931ea4263c
I20260806 09:09:08.768677 15641 ts_tablet_manager.cc:1939] T 865adfc7acff445491ae7e931ea4263c P 1fa0214f7ac24e4fbdb0e74fcfed4d12: Deleting consensus metadata
I20260806 09:09:08.770480 15134 catalog_manager.cc:4841] TS 1fa0214f7ac24e4fbdb0e74fcfed4d12 (127.27.187.131:46511): tablet 865adfc7acff445491ae7e931ea4263c (table test-table [id=6e06fc357c994674b8c980640404f6ec]) successfully deleted
I20260806 09:09:08.882928 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15207
W20260806 09:09:08.904094 15133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:43007 (error 108)
I20260806 09:09:08.904241 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15338
I20260806 09:09:08.906318 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.905609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48220)
0806 09:09:08.905925 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.906075 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 09:09:08.906652 15133 catalog_manager.cc:4584] TS 4356e24703b44e26ad260c5269241158 (127.27.187.129:43007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 865adfc7acff445491ae7e931ea4263c: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
I20260806 09:09:08.912714 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.905182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.27.187.130:33707)
0806 09:09:08.905366 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:08.905373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:08.905397 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:09:08.905596 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:08.905602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:08.905740 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:08.905868 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:08.906607 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.907638 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.909167 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:08.909720 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:08.909850 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:08.911093 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:08.911104 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:08.911122 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:08.911869 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:08.911926 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:08.911949 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:08.912076 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:08.912292 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 09:09:08.912425 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:09:08.913513 15461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:08.960880 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.960332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48236)
0806 09:09:08.960599 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.960711 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 09:09:08.962203 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:08.961713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48238)
0806 09:09:08.961900 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:08.961990 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:09:09.010804 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15338
I20260806 09:09:09.023039 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:09.022360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48246)
0806 09:09:09.022652 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:09.022817 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:09:09.024333 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:09.023911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48254)
0806 09:09:09.024085 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:09.024188 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:09:09.031602 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15470
I20260806 09:09:09.039361 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:09.032490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.27.187.131:46511)
0806 09:09:09.032733 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:09.032738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:09.032757 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:09:09.032821 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:09.032825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:09.032921 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:09.033017 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:09.033514 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:09.034357 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:09.036128 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:09.036602 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:09.036716 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:09.037926 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:09.037936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:09.037952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:09.038550 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:09.038615 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:09.038637 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:09.038822 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:09.038955 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 09:09:09.039062 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:09:09.040071 15593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:09.105172 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:09.104521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48262)
0806 09:09:09.104885 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:09.104998 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260806 09:09:09.106344 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:09.105935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43007 (local address 127.0.0.1:48278)
0806 09:09:09.106150 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:09.106225 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 09:09:09.134279 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15470
I20260806 09:09:09.153236 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15116
I20260806 09:09:09.154495 15177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:09.270951 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15116
2026-08-06T09:09:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:09 dist-test-slave-8tdl krb5kdc[15652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:09 dist-test-slave-8tdl krb5kdc[15652](info): set up 2 sockets
Aug 06 09:09:09 dist-test-slave-8tdl krb5kdc[15652](info): commencing operation
krb5kdc: starting...
W20260806 09:09:10.445888 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:10 dist-test-slave-8tdl krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:10.816239 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34781
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:11.204487 15665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:11.204850 15665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:11.205014 15665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:11.212936 15665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:11.213073 15665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:11.213171 15665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:11.213260 15665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:11.213339 15665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:11.224730 15665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:11.226306 15665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:11.227775 15665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:11.240985 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:11.241115 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:11.241231 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:11.241928 15665 server_base.cc:1034] running on GCE node
I20260806 09:09:11.242882 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:11.244509 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:11.245769 15665 hybrid_clock.cc:648] HybridClock initialized: now 1786007351245708 us; error 44 us; skew 500 ppm
I20260806 09:09:11.246206 15665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:11 dist-test-slave-8tdl krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:11.250072 15665 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:11.252017 15665 webserver.cc:460] Webserver started at http://127.27.187.190:41103/ using document root <none> and password file <none>
I20260806 09:09:11.252843 15665 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:11.252988 15665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:11.253347 15665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:11.256843 15665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "2a3683325fe9416484a107b965eab4e0"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "52a2fc0e95d4ba9170c53ac74e2da510"
server_key_iv: "cc03345f96081d4f97cc2411e09b0c0e"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.257725 15665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "2a3683325fe9416484a107b965eab4e0"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "52a2fc0e95d4ba9170c53ac74e2da510"
server_key_iv: "cc03345f96081d4f97cc2411e09b0c0e"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.263870 15665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:09:11.268280 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:11.269912 15665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:09:11.270103 15665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "2a3683325fe9416484a107b965eab4e0"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "52a2fc0e95d4ba9170c53ac74e2da510"
server_key_iv: "cc03345f96081d4f97cc2411e09b0c0e"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.270382 15665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:11.311436 15665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:11.317122 15665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:11.317497 15665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:11.339649 15665 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34781
I20260806 09:09:11.339672 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34781 every 8 connection(s)
I20260806 09:09:11.341315 15665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:11.343562 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15665
I20260806 09:09:11.343777 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:11.344388 28398 external_mini_cluster.cc:1428] Setting key 7888d624bffe90bb5aef10ed64078f3a
I20260806 09:09:11.346144 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:11.361327 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: Bootstrap starting.
Aug 06 09:09:11 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:11.365634 15733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:11.367148 15733 log.cc:824] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:11.371224 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: No bootstrap required, opened a new log
I20260806 09:09:11.373303 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:11.346636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.27.187.190:34781)
0806 09:09:11.347595 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:11.347611 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:11.347666 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:09:11.349885 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:11.349918 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:11.350548 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:11.351107 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:11.352920 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:11.355146 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:11.356771 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:11.357920 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:11.358150 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:11.363253 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:11.363268 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:11.363298 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:11.363373 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:11.367675 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:11.368834 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:11.368841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:11.368848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:11.369115 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:11.369624 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:11.369629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:11.369634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:11.370418 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:11.370774 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:11.371273 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 09:09:11.371851 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":363,"threads_started":1}
I20260806 09:09:11.381121 15733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } }
I20260806 09:09:11.381562 15733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:11.381644 15733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3683325fe9416484a107b965eab4e0, State: Initialized, Role: FOLLOWER
I20260806 09:09:11.382216 15733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } }
I20260806 09:09:11.382444 15733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:11.382548 15733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:11.382766 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:11.385147 15733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } }
I20260806 09:09:11.385720 15733 leader_election.cc:302] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a3683325fe9416484a107b965eab4e0; no voters:
I20260806 09:09:11.386528 15733 leader_election.cc:288] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:11.386742 15738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:11.388206 15738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a3683325fe9416484a107b965eab4e0, State: Running, Role: LEADER
I20260806 09:09:11.388774 15738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } }
I20260806 09:09:11.389019 15733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:11.391697 15740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3683325fe9416484a107b965eab4e0. Latest consensus state: current_term: 1 leader_uuid: "2a3683325fe9416484a107b965eab4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } } }
I20260806 09:09:11.392174 15740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:11.392752 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a3683325fe9416484a107b965eab4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3683325fe9416484a107b965eab4e0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34781 } } }
I20260806 09:09:11.393034 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:11.394917 15745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:11.400949 15745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:11.413085 15745 catalog_manager.cc:1347] Generated new cluster ID: 367354836c934c1da86c558ff18efdc7
I20260806 09:09:11.413331 15745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:11.427590 15745 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:11.429008 15745 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:11.438764 15745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: Generated new TSK 0
I20260806 09:09:11.439771 15745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.187.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
Entry for principal kudu/127.27.187.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab.
I20260806 09:09:11.519338 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:11.898272 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:11.898610 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:11.898782 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:11.906481 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:11.906621 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:11.906837 15759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:09:11.917498 15759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.129.keytab
--principal=kudu/127.27.187.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:11.918936 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:11.920305 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:11.935828 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:11.935866 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:11.935921 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:11.936685 15759 server_base.cc:1034] running on GCE node
I20260806 09:09:11.937542 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:11.938869 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:11.940142 15759 hybrid_clock.cc:648] HybridClock initialized: now 1786007351940085 us; error 42 us; skew 500 ppm
I20260806 09:09:11.940671 15759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:11 dist-test-slave-8tdl krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:11.944638 15759 init.cc:377] Logged in from keytab as kudu/127.27.187.129@KRBTEST.COM (short username kudu)
I20260806 09:09:11.946450 15759 webserver.cc:460] Webserver started at http://127.27.187.129:43149/ using document root <none> and password file <none>
I20260806 09:09:11.947320 15759 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:11.947445 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:11.947736 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:11.951154 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "eb651154cd664adaa170051308e3cebb"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "55e33789d929f8d12dc1ba7e1d271715"
server_key_iv: "23d3b71a4b504809c37380e9ff70d126"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.952474 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "eb651154cd664adaa170051308e3cebb"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "55e33789d929f8d12dc1ba7e1d271715"
server_key_iv: "23d3b71a4b504809c37380e9ff70d126"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.958750 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:09:11.963295 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:11.965245 15759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:09:11.965502 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "eb651154cd664adaa170051308e3cebb"
format_stamp: "Formatted at 2026-08-06 09:09:11 on dist-test-slave-8tdl"
server_key: "55e33789d929f8d12dc1ba7e1d271715"
server_key_iv: "23d3b71a4b504809c37380e9ff70d126"
server_key_version: "encryptionkey@0"
I20260806 09:09:11.965741 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:11.984265 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:11.989344 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:11.989692 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:11.990856 15759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:11.992408 15759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:11.992486 15759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:11.992666 15759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:11.992731 15759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:12.032852 15759 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:42359
I20260806 09:09:12.032917 15889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:42359 every 8 connection(s)
I20260806 09:09:12.034631 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:09:12.035858 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15759
I20260806 09:09:12.036057 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:09:12.036577 28398 external_mini_cluster.cc:1428] Setting key 7fc91da3f303d2fb07eb9054370d3d3f
Aug 06 09:09:12 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.129@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:12.060652 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:12.038240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:45675 (local address 127.27.187.190:34781)
0806 09:09:12.038823 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:12.038830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:12.040343 (+ 1513us) server_negotiation.cc:407] Connection header received
0806 09:09:12.042553 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:12.042559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:12.042751 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:12.042894 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:12.044782 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.045604 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.047609 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.048083 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.048252 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:12.053087 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:12.053094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:12.053098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:12.053146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:12.056523 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:12.057634 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:12.057642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:12.057645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:12.057806 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:12.058382 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:12.058388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:12.058390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:12.058822 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:12.059012 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:12.059903 (+ 891us) server_negotiation.cc:299] Negotiation successful
0806 09:09:12.060205 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260806 09:09:12.061831 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:12.039098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34781 (local address 127.27.187.129:45675)
0806 09:09:12.040089 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:12.040182 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:12.042211 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:12.043201 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:12.043210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:12.043854 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:12.044549 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:12.044572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.045817 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.045822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:12.047410 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:12.047425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.048382 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.048391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:12.048669 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:12.049581 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:12.049616 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:12.052747 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:12.056779 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:12.056799 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:12.056817 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:12.057426 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:12.057952 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:12.057957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:12.057961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:12.058216 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:12.058982 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:12.058993 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:12.059439 (+ 446us) client_negotiation.cc:769] Sending connection context
0806 09:09:12.059780 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 09:09:12.060442 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":385,"threads_started":1}
I20260806 09:09:12.063867 15890 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34781
I20260806 09:09:12.064226 15890 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:12.065039 15890 heartbeater.cc:511] Master 127.27.187.190:34781 requested a full tablet report, sending...
I20260806 09:09:12.067412 15697 ts_manager.cc:194] Registered new tserver with Master: eb651154cd664adaa170051308e3cebb (127.27.187.129:42359)
I20260806 09:09:12.069010 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.129@KRBTEST.COM'} at 127.27.187.129:45675
WARNING: no policy specified for kudu/127.27.187.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
Entry for principal kudu/127.27.187.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab.
I20260806 09:09:12.116492 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:12.477494 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:12.477828 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:12.477996 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:12.485893 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:12.486034 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:12.486207 15896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:09:12.497081 15896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.130.keytab
--principal=kudu/127.27.187.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:12.498545 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:12.500088 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:12.514794 15901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:12.514828 15902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:12.514871 15904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:12.516106 15896 server_base.cc:1034] running on GCE node
I20260806 09:09:12.516887 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:12.518009 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:12.519260 15896 hybrid_clock.cc:648] HybridClock initialized: now 1786007352519282 us; error 133 us; skew 500 ppm
I20260806 09:09:12.519622 15896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:12 dist-test-slave-8tdl krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:12.523365 15896 init.cc:377] Logged in from keytab as kudu/127.27.187.130@KRBTEST.COM (short username kudu)
I20260806 09:09:12.525211 15896 webserver.cc:460] Webserver started at http://127.27.187.130:35835/ using document root <none> and password file <none>
I20260806 09:09:12.526010 15896 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:12.526173 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:12.526574 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:12.530207 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "db4666483340434b92fc078b7c87c58e"
format_stamp: "Formatted at 2026-08-06 09:09:12 on dist-test-slave-8tdl"
server_key: "51dabda035c71eaf344713c1369eceeb"
server_key_iv: "685bb79da204acf7078f39c4c11c4ce7"
server_key_version: "encryptionkey@0"
I20260806 09:09:12.531178 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "db4666483340434b92fc078b7c87c58e"
format_stamp: "Formatted at 2026-08-06 09:09:12 on dist-test-slave-8tdl"
server_key: "51dabda035c71eaf344713c1369eceeb"
server_key_iv: "685bb79da204acf7078f39c4c11c4ce7"
server_key_version: "encryptionkey@0"
I20260806 09:09:12.537537 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:12.541889 15911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:12.543663 15896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:12.543875 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "db4666483340434b92fc078b7c87c58e"
format_stamp: "Formatted at 2026-08-06 09:09:12 on dist-test-slave-8tdl"
server_key: "51dabda035c71eaf344713c1369eceeb"
server_key_iv: "685bb79da204acf7078f39c4c11c4ce7"
server_key_version: "encryptionkey@0"
I20260806 09:09:12.544119 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:12.568321 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:12.573568 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:12.573904 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:12.575124 15896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:12.576728 15896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:12.576848 15896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:12.576997 15896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:12.577070 15896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:12.615638 15896 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:38283
I20260806 09:09:12.615707 16025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:38283 every 8 connection(s)
I20260806 09:09:12.617103 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:09:12.623190 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 15896
I20260806 09:09:12.623397 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:09:12.623973 28398 external_mini_cluster.cc:1428] Setting key 7bf0978a1fed34851e6d39eb1cb4e4c1
Aug 06 09:09:12 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.130@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:12.644099 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:12.620305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:55073 (local address 127.27.187.190:34781)
0806 09:09:12.620896 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:12.620903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:12.621903 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 09:09:12.624627 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:12.624634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:12.624737 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:12.624876 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:12.627279 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.628061 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.630843 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.631299 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.631401 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:12.636673 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:12.636680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:12.636683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:12.636727 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:12.639778 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:12.640949 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:12.640958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:12.640963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:12.641151 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:12.641803 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:12.641812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:12.641816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:12.642258 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:12.642439 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:12.643277 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 09:09:12.643568 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":298,"threads_started":1}
I20260806 09:09:12.644940 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:12.620869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34781 (local address 127.27.187.130:55073)
0806 09:09:12.621704 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:12.621744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:12.624217 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:12.625276 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:12.625285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:12.626010 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:12.626981 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:12.627015 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.628257 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.628266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:12.630619 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:12.630640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:12.631556 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:12.631566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:12.631923 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:12.632809 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:12.632854 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:12.636267 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:12.639972 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:12.639986 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:12.640010 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:12.640747 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:12.641415 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:12.641420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:12.641422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:12.641645 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:12.642399 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:12.642405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:12.642868 (+ 463us) client_negotiation.cc:769] Sending connection context
0806 09:09:12.643114 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 09:09:12.643601 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":332,"threads_started":1}
I20260806 09:09:12.646653 16026 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34781
I20260806 09:09:12.647070 16026 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:12.647862 16026 heartbeater.cc:511] Master 127.27.187.190:34781 requested a full tablet report, sending...
I20260806 09:09:12.649688 15697 ts_manager.cc:194] Registered new tserver with Master: db4666483340434b92fc078b7c87c58e (127.27.187.130:38283)
I20260806 09:09:12.651087 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.130@KRBTEST.COM'} at 127.27.187.130:55073
WARNING: no policy specified for kudu/127.27.187.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
Entry for principal kudu/127.27.187.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab.
I20260806 09:09:12.703442 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:09:13.072791 15890 heartbeater.cc:503] Master 127.27.187.190:34781 was elected leader, sending a full tablet report...
W20260806 09:09:13.079121 16032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:13.079505 16032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:13.079700 16032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:13.087760 16032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:13.087953 16032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:13.088147 16032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:09:13.099287 16032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.131.keytab
--principal=kudu/127.27.187.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:34781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:13.100816 16032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:13.102315 16032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:13.117208 16037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:13.117164 16038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:13.117151 16040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:13.118631 16032 server_base.cc:1034] running on GCE node
I20260806 09:09:13.119585 16032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:13.120728 16032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:13.122015 16032 hybrid_clock.cc:648] HybridClock initialized: now 1786007353121937 us; error 70 us; skew 500 ppm
I20260806 09:09:13.122545 16032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:13 dist-test-slave-8tdl krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:13.131872 16032 init.cc:377] Logged in from keytab as kudu/127.27.187.131@KRBTEST.COM (short username kudu)
I20260806 09:09:13.133767 16032 webserver.cc:460] Webserver started at http://127.27.187.131:42759/ using document root <none> and password file <none>
I20260806 09:09:13.134629 16032 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:13.134857 16032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:13.135236 16032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:13.138559 16032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "33ebbce4bc6f4de5852a262c15916c36"
format_stamp: "Formatted at 2026-08-06 09:09:13 on dist-test-slave-8tdl"
server_key: "7550308b4f4e1eb5c5040e2e1057d5fe"
server_key_iv: "2ba19be893dd705f3f40b421fab83569"
server_key_version: "encryptionkey@0"
I20260806 09:09:13.139521 16032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "33ebbce4bc6f4de5852a262c15916c36"
format_stamp: "Formatted at 2026-08-06 09:09:13 on dist-test-slave-8tdl"
server_key: "7550308b4f4e1eb5c5040e2e1057d5fe"
server_key_iv: "2ba19be893dd705f3f40b421fab83569"
server_key_version: "encryptionkey@0"
I20260806 09:09:13.145462 16032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:09:13.149771 16047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:13.151408 16032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:13.151651 16032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "33ebbce4bc6f4de5852a262c15916c36"
format_stamp: "Formatted at 2026-08-06 09:09:13 on dist-test-slave-8tdl"
server_key: "7550308b4f4e1eb5c5040e2e1057d5fe"
server_key_iv: "2ba19be893dd705f3f40b421fab83569"
server_key_version: "encryptionkey@0"
I20260806 09:09:13.151886 16032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:13.164875 16032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:13.169911 16032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:13.170260 16032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:13.171464 16032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:13.172919 16032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:13.173025 16032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:13.173182 16032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:13.173271 16032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:13.213752 16032 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:46647
I20260806 09:09:13.213802 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:46647 every 8 connection(s)
I20260806 09:09:13.215286 16032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:09:13.220542 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16032
I20260806 09:09:13.220760 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:09:13.221237 28398 external_mini_cluster.cc:1428] Setting key 5f7a1aa16564349fef2e24043a7dffd4
Aug 06 09:09:13 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.131@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:13.239723 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.218280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:53779 (local address 127.27.187.190:34781)
0806 09:09:13.218841 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.218847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.220033 (+ 1186us) server_negotiation.cc:407] Connection header received
0806 09:09:13.222333 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.222339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.222459 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.222598 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:13.224693 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.225508 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.227410 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.227873 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.227992 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.232316 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:13.232322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:13.232325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:13.232365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:13.235946 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.237003 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.237015 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.237017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.237167 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.237670 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.237674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.237676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.237965 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:13.238152 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.238987 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.239278 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260806 09:09:13.240787 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.218996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:34781 (local address 127.27.187.131:53779)
0806 09:09:13.219834 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.219885 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.222021 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.222975 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.222984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.223744 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:13.224433 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.224462 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.225680 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.225684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.227205 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.227219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.228170 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.228177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.228400 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.229239 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:09:13.229264 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:13.231994 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:13.236129 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:13.236140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:13.236161 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:13.236855 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:13.237304 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:09:13.237309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:09:13.237319 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:09:13.237524 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:09:13.238121 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:13.238129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:13.238532 (+ 403us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.238880 (+ 348us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.239478 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260806 09:09:13.242362 16162 heartbeater.cc:347] Connected to a master server at 127.27.187.190:34781
I20260806 09:09:13.242722 16162 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:13.243448 16162 heartbeater.cc:511] Master 127.27.187.190:34781 requested a full tablet report, sending...
I20260806 09:09:13.244935 15697 ts_manager.cc:194] Registered new tserver with Master: 33ebbce4bc6f4de5852a262c15916c36 (127.27.187.131:46647)
I20260806 09:09:13.245752 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.187.131@KRBTEST.COM'} at 127.27.187.131:53779
I20260806 09:09:13.250325 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:09:13.269593 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.255022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.27.187.190:34781)
0806 09:09:13.255243 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.255248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.255809 (+ 561us) server_negotiation.cc:407] Connection header received
0806 09:09:13.255994 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.255998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.256068 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.256164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:13.257770 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.258559 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.259987 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.260475 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.260582 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.263417 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:13.263423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:13.263426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:13.263467 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:13.266447 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.267412 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.267417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.267419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.267583 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.268056 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.268060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.268062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.268467 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:13.268674 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.269092 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.269354 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:09:13.274765 15697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:09:13.282155 15697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:09:13.307473 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.296183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38283 (local address 127.0.0.1:60198)
0806 09:09:13.297635 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.297674 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.297864 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.299776 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.299784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.299812 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:13.300276 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.300291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.302282 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.302288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.304662 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.304675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.306348 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.306355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.306581 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.306644 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.306830 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.307036 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":980,"threads_started":1}
I20260806 09:09:13.307615 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.296438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.27.187.130:38283)
0806 09:09:13.298848 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.298856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.298880 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:09:13.298970 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.298975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.299253 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.299462 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:13.300476 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.302116 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.304843 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.306100 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.306298 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.306502 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.307100 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.307310 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":2115,"threads_started":1}
I20260806 09:09:13.309252 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.296455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:46647 (local address 127.0.0.1:45002)
0806 09:09:13.298256 (+ 1801us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.298271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.298356 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.300752 (+ 2396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.300759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.300800 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:13.301375 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.301393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.303441 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.303448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.305722 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.305737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.308307 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.308313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.308491 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.308526 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.308602 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.308760 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"mutex_wait_us":24,"thread_start_us":1511,"threads_started":1}
I20260806 09:09:13.310482 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.297165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.27.187.131:46647)
0806 09:09:13.300081 (+ 2916us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.300091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.300127 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:09:13.300235 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.300244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.300492 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.301495 (+ 1003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:13.301791 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.303253 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.305890 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.308053 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.309599 (+ 1546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.309812 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.309899 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.310102 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2713,"thread_start_us":3271,"threads_started":1}
I20260806 09:09:13.311260 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.299822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:52962)
0806 09:09:13.301187 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.301208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.301322 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.302709 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.302715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.302743 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:13.303477 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.303493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.305606 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.305611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.307802 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.307820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.310162 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.310170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.310359 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.310395 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.310491 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.310860 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":1,"thread_start_us":1031,"threads_started":1}
I20260806 09:09:13.311453 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.300473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.27.187.129:42359)
0806 09:09:13.302061 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.302067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.302089 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:09:13.302162 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.302167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.302442 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.302652 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:13.303753 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.305444 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.308066 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.309918 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.310134 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.310314 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.310918 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.311101 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":1251,"threads_started":1}
I20260806 09:09:13.312497 15959 tablet_service.cc:1467] Processing CreateTablet for tablet e8d7d4b809634008842e7c871bdd5493 (DEFAULT_TABLE table=test-table [id=8eac0e199dd24c1bae7d11c0f4c826a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:13.315378 15823 tablet_service.cc:1467] Processing CreateTablet for tablet e8d7d4b809634008842e7c871bdd5493 (DEFAULT_TABLE table=test-table [id=8eac0e199dd24c1bae7d11c0f4c826a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:13.315378 16095 tablet_service.cc:1467] Processing CreateTablet for tablet e8d7d4b809634008842e7c871bdd5493 (DEFAULT_TABLE table=test-table [id=8eac0e199dd24c1bae7d11c0f4c826a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:13.316234 15959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d7d4b809634008842e7c871bdd5493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:13.316879 16095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d7d4b809634008842e7c871bdd5493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:13.316874 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d7d4b809634008842e7c871bdd5493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:13.331157 16181 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Bootstrap starting.
I20260806 09:09:13.332695 16182 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Bootstrap starting.
I20260806 09:09:13.335476 16183 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Bootstrap starting.
I20260806 09:09:13.336380 16182 tablet_bootstrap.cc:654] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:13.336787 16181 tablet_bootstrap.cc:654] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:13.337798 16182 log.cc:824] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:13.338132 16181 log.cc:824] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:13.338918 16183 tablet_bootstrap.cc:654] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:13.340150 16183 log.cc:824] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:13.341365 16182 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: No bootstrap required, opened a new log
I20260806 09:09:13.341645 16182 ts_tablet_manager.cc:1397] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 09:09:13.343333 16183 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: No bootstrap required, opened a new log
I20260806 09:09:13.343636 16183 ts_tablet_manager.cc:1397] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260806 09:09:13.344189 16181 tablet_bootstrap.cc:492] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: No bootstrap required, opened a new log
I20260806 09:09:13.344520 16181 ts_tablet_manager.cc:1397] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20260806 09:09:13.352854 16182 raft_consensus.cc:348] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.353444 16182 raft_consensus.cc:374] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:13.353556 16182 raft_consensus.cc:729] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb651154cd664adaa170051308e3cebb, State: Initialized, Role: FOLLOWER
I20260806 09:09:13.354422 16182 consensus_queue.cc:260] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.355089 16183 raft_consensus.cc:348] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.355610 16183 raft_consensus.cc:374] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:13.355723 16183 raft_consensus.cc:729] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebbce4bc6f4de5852a262c15916c36, State: Initialized, Role: FOLLOWER
I20260806 09:09:13.356506 16183 consensus_queue.cc:260] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.356769 16181 raft_consensus.cc:348] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.357661 16181 raft_consensus.cc:374] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:13.357841 16181 raft_consensus.cc:729] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4666483340434b92fc078b7c87c58e, State: Initialized, Role: FOLLOWER
I20260806 09:09:13.358872 16181 consensus_queue.cc:260] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.365597 16183 ts_tablet_manager.cc:1428] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Time spent starting tablet: real 0.022s user 0.014s sys 0.006s
I20260806 09:09:13.366122 16162 heartbeater.cc:503] Master 127.27.187.190:34781 was elected leader, sending a full tablet report...
I20260806 09:09:13.371747 16182 ts_tablet_manager.cc:1428] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Time spent starting tablet: real 0.030s user 0.024s sys 0.008s
I20260806 09:09:13.373729 16026 heartbeater.cc:503] Master 127.27.187.190:34781 was elected leader, sending a full tablet report...
I20260806 09:09:13.374405 16181 ts_tablet_manager.cc:1428] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Time spent starting tablet: real 0.030s user 0.013s sys 0.015s
I20260806 09:09:13.412865 16187 raft_consensus.cc:482] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:09:13.413275 16187 raft_consensus.cc:504] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.415233 16187 leader_election.cc:288] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db4666483340434b92fc078b7c87c58e (127.27.187.130:38283), 33ebbce4bc6f4de5852a262c15916c36 (127.27.187.131:46647)
I20260806 09:09:13.425726 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.415758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:47029 (local address 127.27.187.130:38283)
0806 09:09:13.415976 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.415983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.417348 (+ 1365us) server_negotiation.cc:407] Connection header received
0806 09:09:13.417437 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.417441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.417532 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.417625 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:13.418472 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.419655 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.423029 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.424404 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.424594 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.424677 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.425311 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.425454 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:09:13.425768 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.415758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:59235 (local address 127.27.187.131:46647)
0806 09:09:13.415976 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.415983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.416998 (+ 1015us) server_negotiation.cc:407] Connection header received
0806 09:09:13.417325 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.417332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.417423 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.417540 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:13.418409 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.419693 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.423020 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.424404 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.424594 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.424679 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.425300 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.425446 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:09:13.425935 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.415622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:46647 (local address 127.27.187.129:59235)
0806 09:09:13.416801 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.416828 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.417070 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.417628 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.417636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.417671 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:13.418170 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.418184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.419857 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.419864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.422777 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.422798 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.424752 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.424761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.425103 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.425131 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.425211 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.425414 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":189,"thread_start_us":1049,"threads_started":1}
I20260806 09:09:13.426545 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.415622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:38283 (local address 127.27.187.129:47029)
0806 09:09:13.417199 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.417217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:13.417343 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:13.417725 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:13.417731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:13.417824 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:13.418281 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.418294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.419857 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.419864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.422780 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:13.422798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.424771 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.424777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:13.425048 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.425088 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:09:13.425212 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 09:09:13.425414 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"mutex_wait_us":412,"spinlock_wait_cycles":27520,"thread_start_us":600,"threads_started":1}
I20260806 09:09:13.427227 16115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d7d4b809634008842e7c871bdd5493" candidate_uuid: "eb651154cd664adaa170051308e3cebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ebbce4bc6f4de5852a262c15916c36" is_pre_election: true
I20260806 09:09:13.427650 15979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d7d4b809634008842e7c871bdd5493" candidate_uuid: "eb651154cd664adaa170051308e3cebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4666483340434b92fc078b7c87c58e" is_pre_election: true
I20260806 09:09:13.427776 16115 raft_consensus.cc:2454] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb651154cd664adaa170051308e3cebb in term 0.
I20260806 09:09:13.428222 15979 raft_consensus.cc:2454] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb651154cd664adaa170051308e3cebb in term 0.
I20260806 09:09:13.428536 15777 leader_election.cc:302] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ebbce4bc6f4de5852a262c15916c36, eb651154cd664adaa170051308e3cebb; no voters:
I20260806 09:09:13.429004 16187 raft_consensus.cc:2781] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:09:13.429189 16187 raft_consensus.cc:482] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:09:13.429323 16187 raft_consensus.cc:3037] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:13.431926 16187 raft_consensus.cc:504] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.432646 16187 leader_election.cc:288] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [CANDIDATE]: Term 1 election: Requested vote from peers db4666483340434b92fc078b7c87c58e (127.27.187.130:38283), 33ebbce4bc6f4de5852a262c15916c36 (127.27.187.131:46647)
I20260806 09:09:13.433362 15979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d7d4b809634008842e7c871bdd5493" candidate_uuid: "eb651154cd664adaa170051308e3cebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4666483340434b92fc078b7c87c58e"
I20260806 09:09:13.433363 16115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d7d4b809634008842e7c871bdd5493" candidate_uuid: "eb651154cd664adaa170051308e3cebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ebbce4bc6f4de5852a262c15916c36"
I20260806 09:09:13.433604 15979 raft_consensus.cc:3037] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:13.433625 16115 raft_consensus.cc:3037] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:13.437088 15979 raft_consensus.cc:2454] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb651154cd664adaa170051308e3cebb in term 1.
I20260806 09:09:13.437088 16115 raft_consensus.cc:2454] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb651154cd664adaa170051308e3cebb in term 1.
I20260806 09:09:13.437680 15777 leader_election.cc:302] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33ebbce4bc6f4de5852a262c15916c36, db4666483340434b92fc078b7c87c58e, eb651154cd664adaa170051308e3cebb; no voters:
I20260806 09:09:13.438024 16187 raft_consensus.cc:2781] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:13.438530 16187 raft_consensus.cc:686] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 1 LEADER]: Becoming Leader. State: Replica: eb651154cd664adaa170051308e3cebb, State: Running, Role: LEADER
I20260806 09:09:13.439165 16187 consensus_queue.cc:237] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } }
I20260806 09:09:13.444133 15696 catalog_manager.cc:5495] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb reported cstate change: term changed from 0 to 1, leader changed from <none> to eb651154cd664adaa170051308e3cebb (127.27.187.129). New cstate: current_term: 1 leader_uuid: "eb651154cd664adaa170051308e3cebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb651154cd664adaa170051308e3cebb" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 42359 } health_report { overall_health: HEALTHY } } }
I20260806 09:09:13.465631 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.459367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.27.187.129:42359)
0806 09:09:13.459577 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.459582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.459601 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:09:13.459880 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.459884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.459984 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.460202 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:09:13.460504 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.461300 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.462982 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.463498 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.463612 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.464391 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:09:13.464984 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:09:13.465109 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.465318 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.465404 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:09:13.467268 16163 tablet.cc:2369] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:09:13.475910 16115 raft_consensus.cc:1261] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 1 FOLLOWER]: Refusing update from remote peer eb651154cd664adaa170051308e3cebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:13.475911 15979 raft_consensus.cc:1261] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 1 FOLLOWER]: Refusing update from remote peer eb651154cd664adaa170051308e3cebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:13.477398 16192 consensus_queue.cc:1035] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebbce4bc6f4de5852a262c15916c36" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:13.477869 16187 consensus_queue.cc:1035] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "db4666483340434b92fc078b7c87c58e" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:13.490944 16195 mvcc.cc:205] Tried to move back new op lower bound from 7315486119823736832 to 7315486119690792960. Current Snapshot: MvccSnapshot[applied={T|T < 7315486119823736832}]
I20260806 09:09:13.508991 15696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:13.509366 15696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:13.514014 15696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3683325fe9416484a107b965eab4e0: Sending DeleteTablet for 3 replicas of tablet e8d7d4b809634008842e7c871bdd5493
I20260806 09:09:13.515369 15959 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d7d4b809634008842e7c871bdd5493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:60198
I20260806 09:09:13.515448 15823 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d7d4b809634008842e7c871bdd5493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:52962
I20260806 09:09:13.515561 16095 tablet_service.cc:1514] Processing DeleteTablet for tablet e8d7d4b809634008842e7c871bdd5493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:13 UTC) from {username='kudu', principal='kudu/127.27.187.190@KRBTEST.COM'} at 127.0.0.1:45002
I20260806 09:09:13.517827 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15759
I20260806 09:09:13.546862 16204 tablet_replica.cc:330] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: stopping tablet replica
I20260806 09:09:13.547554 16204 raft_consensus.cc:2229] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:13.552593 16204 raft_consensus.cc:2258] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:13.558169 16204 ts_tablet_manager.cc:1905] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:09:13 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.129@KRBTEST.COM
I20260806 09:09:13.569936 16206 tablet_replica.cc:330] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: stopping tablet replica
I20260806 09:09:13.570725 16206 raft_consensus.cc:2229] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:13.570922 16205 tablet_replica.cc:330] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: stopping tablet replica
I20260806 09:09:13.571573 16205 raft_consensus.cc:2229] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:09:13.571666 16206 raft_consensus.cc:2258] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:13.572947 16205 raft_consensus.cc:2258] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:13.574324 16204 ts_tablet_manager.cc:1918] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:13.574515 16204 log.cc:1196] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/e8d7d4b809634008842e7c871bdd5493
I20260806 09:09:13.575356 16206 ts_tablet_manager.cc:1905] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:13.576411 16204 ts_tablet_manager.cc:1939] T e8d7d4b809634008842e7c871bdd5493 P db4666483340434b92fc078b7c87c58e: Deleting consensus metadata
I20260806 09:09:13.577008 16205 ts_tablet_manager.cc:1905] T e8d7d4b809634008842e7c871bdd5493 P eb651154cd664adaa170051308e3cebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:13.579319 15683 catalog_manager.cc:4841] TS db4666483340434b92fc078b7c87c58e (127.27.187.130:38283): tablet e8d7d4b809634008842e7c871bdd5493 (table test-table [id=8eac0e199dd24c1bae7d11c0f4c826a6]) successfully deleted
I20260806 09:09:13.579994 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.519144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.27.187.129:42359)
0806 09:09:13.519385 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.519392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.543205 (+ 23813us) server_negotiation.cc:407] Connection header received
0806 09:09:13.543827 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.543836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.543937 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.544027 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:13.548313 (+ 4286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.549795 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.555814 (+ 6019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.556663 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.556823 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.571326 (+ 14503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:13.571348 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:13.571367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:13.571436 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:13.575951 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.577074 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.577084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.577088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.577268 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.577788 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.577793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.577796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.578318 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:13.578651 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.579330 (+ 679us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.579641 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:09:13.581974 15885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:13.584823 16206 ts_tablet_manager.cc:1918] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:13.585129 16206 log.cc:1196] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/e8d7d4b809634008842e7c871bdd5493
I20260806 09:09:13.585760 16206 ts_tablet_manager.cc:1939] T e8d7d4b809634008842e7c871bdd5493 P 33ebbce4bc6f4de5852a262c15916c36: Deleting consensus metadata
I20260806 09:09:13.587966 15682 catalog_manager.cc:4841] TS 33ebbce4bc6f4de5852a262c15916c36 (127.27.187.131:46647): tablet e8d7d4b809634008842e7c871bdd5493 (table test-table [id=8eac0e199dd24c1bae7d11c0f4c826a6]) successfully deleted
I20260806 09:09:13.690768 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15759
I20260806 09:09:13.712510 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15896
W20260806 09:09:13.712680 15682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.187.129:42359 (error 108)
I20260806 09:09:13.714663 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.714209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:52984)
0806 09:09:13.714390 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.714502 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 09:09:13.715010 15682 catalog_manager.cc:4584] TS eb651154cd664adaa170051308e3cebb (127.27.187.129:42359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8d7d4b809634008842e7c871bdd5493: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Aug 06 09:09:13 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.130@KRBTEST.COM
I20260806 09:09:13.732142 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.713523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.27.187.130:38283)
0806 09:09:13.713766 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.713773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.713800 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:09:13.714034 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.714040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.714124 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.714240 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:13.715907 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.716749 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.718127 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.718577 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.718759 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.723465 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:13.723482 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:13.723498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:13.723543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:13.728129 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.729019 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.729029 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.729033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.729242 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.729773 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.729779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.729782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.730405 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:13.730785 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.731502 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.731738 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:09:13.732929 16021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:13.768932 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.768394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:52986)
0806 09:09:13.768645 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.768758 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 09:09:13.770175 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.769725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:52992)
0806 09:09:13.769895 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.770006 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:09:13.830981 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.830452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:53000)
0806 09:09:13.830711 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.830820 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 09:09:13.832217 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.831833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:53010)
0806 09:09:13.831991 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.832093 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:09:13.835497 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15896
I20260806 09:09:13.854758 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16032
Aug 06 09:09:13 dist-test-slave-8tdl krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.131@KRBTEST.COM
I20260806 09:09:13.873361 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.855642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.27.187.131:46647)
0806 09:09:13.855836 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:13.855841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:13.855934 (+ 93us) server_negotiation.cc:407] Connection header received
0806 09:09:13.856283 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:13.856289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:13.856367 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:13.856466 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:13.858093 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.859075 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.860589 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:13.861015 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:13.861124 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:13.866480 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:13.866498 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:13.866515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:13.866567 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:13.869859 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.870876 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.870886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.870888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.871069 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:13.871550 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:13.871554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:13.871556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:13.871968 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:13.872257 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:13.872614 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 09:09:13.872851 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:09:13.874173 16157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:13.913233 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.912663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:53026)
0806 09:09:13.912931 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.913027 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 09:09:13.914776 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:13.914197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:53042)
0806 09:09:13.914441 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:13.914535 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 09:09:13.969820 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16032
I20260806 09:09:13.989923 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 15665
I20260806 09:09:13.991291 15728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:14.110296 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:14.109507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:42359 (local address 127.0.0.1:53056)
0806 09:09:14.109777 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:14.109895 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.129:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:09:14.110423 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 15665
2026-08-06T09:09:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4841 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T09:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:14Z Disabled control of system clock
I20260806 09:09:14.215412 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43333
--webserver_interface=127.27.187.190
--webserver_port=0
--builtin_ntp_servers=127.27.187.148:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:09:14.593309 16216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:14.593662 16216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:14.593750 16216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:14.602243 16216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:14.602404 16216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:14.602445 16216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:14.602497 16216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:14.602545 16216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:14.612509 16216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:43333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:14.613884 16216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:14.615859 16216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:14.630899 16221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:14.630956 16224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:14.630952 16222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:14.632279 16216 server_base.cc:1034] running on GCE node
I20260806 09:09:14.633250 16216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:14.634963 16216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:14.636293 16216 hybrid_clock.cc:648] HybridClock initialized: now 1786007354636281 us; error 84 us; skew 500 ppm
I20260806 09:09:14.636720 16216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:14.639675 16216 webserver.cc:460] Webserver started at http://127.27.187.190:39447/ using document root <none> and password file <none>
I20260806 09:09:14.640460 16216 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:14.640601 16216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:14.640949 16216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:14.644349 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "0de70cfb23514e698a33012863e4ea3c"
format_stamp: "Formatted at 2026-08-06 09:09:14 on dist-test-slave-8tdl"
server_key: "f88121de7cd630a58a56834168325c2c"
server_key_iv: "efb8c209c85b9544047d48d22849eb9f"
server_key_version: "encryptionkey@0"
I20260806 09:09:14.645272 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "0de70cfb23514e698a33012863e4ea3c"
format_stamp: "Formatted at 2026-08-06 09:09:14 on dist-test-slave-8tdl"
server_key: "f88121de7cd630a58a56834168325c2c"
server_key_iv: "efb8c209c85b9544047d48d22849eb9f"
server_key_version: "encryptionkey@0"
I20260806 09:09:14.651373 16216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:09:14.655601 16230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:14.657526 16216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:09:14.657783 16216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "0de70cfb23514e698a33012863e4ea3c"
format_stamp: "Formatted at 2026-08-06 09:09:14 on dist-test-slave-8tdl"
server_key: "f88121de7cd630a58a56834168325c2c"
server_key_iv: "efb8c209c85b9544047d48d22849eb9f"
server_key_version: "encryptionkey@0"
I20260806 09:09:14.658062 16216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:14.697099 16216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:14.698110 16216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:14.698451 16216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:14.719564 16216 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:43333
I20260806 09:09:14.719666 16281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:43333 every 8 connection(s)
I20260806 09:09:14.721187 16216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:14.721732 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16216
I20260806 09:09:14.721932 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:14.722693 28398 external_mini_cluster.cc:1428] Setting key d2ab0bf456fc1a8fa07ca96b42187606
I20260806 09:09:14.726029 16282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:14.737802 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:14.724969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.27.187.190:43333)
0806 09:09:14.726059 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:14.726071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:14.726153 (+ 82us) server_negotiation.cc:407] Connection header received
0806 09:09:14.727988 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:14.728017 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:14.728543 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:14.729004 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:14.729645 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:14.731144 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:14.732599 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:14.733552 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:14.733768 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:14.735134 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:14.735142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:14.735158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:14.735790 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:14.735885 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:14.735909 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:14.736056 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:14.736336 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 09:09:14.736728 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":385,"threads_started":1}
I20260806 09:09:14.739341 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: Bootstrap starting.
I20260806 09:09:14.743033 16282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:14.744230 16282 log.cc:824] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:14.747954 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: No bootstrap required, opened a new log
I20260806 09:09:14.757427 16282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } }
I20260806 09:09:14.757942 16282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:14.758080 16282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0de70cfb23514e698a33012863e4ea3c, State: Initialized, Role: FOLLOWER
I20260806 09:09:14.758884 16282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } }
I20260806 09:09:14.759141 16282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:14.759277 16282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:14.759425 16282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:14.762060 16282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } }
I20260806 09:09:14.762796 16282 leader_election.cc:302] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0de70cfb23514e698a33012863e4ea3c; no voters:
I20260806 09:09:14.763659 16282 leader_election.cc:288] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:14.763785 16287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:14.764652 16287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [term 1 LEADER]: Becoming Leader. State: Replica: 0de70cfb23514e698a33012863e4ea3c, State: Running, Role: LEADER
I20260806 09:09:14.765287 16287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } }
I20260806 09:09:14.766191 16282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:14.768713 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0de70cfb23514e698a33012863e4ea3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } } }
I20260806 09:09:14.768687 16289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0de70cfb23514e698a33012863e4ea3c. Latest consensus state: current_term: 1 leader_uuid: "0de70cfb23514e698a33012863e4ea3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de70cfb23514e698a33012863e4ea3c" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43333 } } }
I20260806 09:09:14.769519 16289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:14.769519 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:14.771908 16292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:14.777951 16292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:14.787688 16292 catalog_manager.cc:1347] Generated new cluster ID: 8c30af47cf0d4e2480c9652eea233646
I20260806 09:09:14.787832 16292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:14.803602 16292 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:14.805071 16292 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:14.820560 16292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: Generated new TSK 0
I20260806 09:09:14.821552 16292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:14.830199 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.129:0
--local_ip_for_outbound_sockets=127.27.187.129
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--builtin_ntp_servers=127.27.187.148:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:09:15.222661 16306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:15.223100 16306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:15.223260 16306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:15.231565 16306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:15.231714 16306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:15.231896 16306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.129
I20260806 09:09:15.242270 16306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.187.129
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.129
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:15.243790 16306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:15.245920 16306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:15.261435 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:15.261539 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:15.261493 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:15.262853 16306 server_base.cc:1034] running on GCE node
I20260806 09:09:15.263669 16306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:15.264865 16306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:15.266146 16306 hybrid_clock.cc:648] HybridClock initialized: now 1786007355266097 us; error 78 us; skew 500 ppm
I20260806 09:09:15.266574 16306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:15.269809 16306 webserver.cc:460] Webserver started at http://127.27.187.129:46451/ using document root <none> and password file <none>
I20260806 09:09:15.270713 16306 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:15.270882 16306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:15.271377 16306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:15.275015 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data/instance:
uuid: "fae693640f164c3487834302c2da74b8"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "1117d1f4145ce73b5d5b1807ebdcf05b"
server_key_iv: "ff63992d5c0be71dea55f96e2674b3f5"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.275908 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance:
uuid: "fae693640f164c3487834302c2da74b8"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "1117d1f4145ce73b5d5b1807ebdcf05b"
server_key_iv: "ff63992d5c0be71dea55f96e2674b3f5"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.282225 16306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:15.286849 16320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.288662 16306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:09:15.288945 16306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal
uuid: "fae693640f164c3487834302c2da74b8"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "1117d1f4145ce73b5d5b1807ebdcf05b"
server_key_iv: "ff63992d5c0be71dea55f96e2674b3f5"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.289233 16306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:15.322388 16306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:15.323457 16306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:15.323822 16306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:15.324918 16306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:15.326546 16306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:15.326678 16306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.326886 16306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:15.326978 16306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.364441 16306 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.129:43817
I20260806 09:09:15.364550 16433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.129:43817 every 8 connection(s)
I20260806 09:09:15.366080 16306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/data/info.pb
I20260806 09:09:15.368369 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16306
I20260806 09:09:15.368567 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal/instance
I20260806 09:09:15.369093 28398 external_mini_cluster.cc:1428] Setting key 3b3dfbde3e76cd117771322dc1f6da71
I20260806 09:09:15.372507 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.130:0
--local_ip_for_outbound_sockets=127.27.187.130
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--builtin_ntp_servers=127.27.187.148:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:09:15.380924 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:15.369617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.129:39551 (local address 127.27.187.190:43333)
0806 09:09:15.370158 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:15.370163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:15.371755 (+ 1592us) server_negotiation.cc:407] Connection header received
0806 09:09:15.373864 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:15.373870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:15.373972 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:15.374113 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:15.375379 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.376128 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.377779 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.378228 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.378376 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:15.379967 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:15.379974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:15.379979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:15.380009 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:15.380052 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:15.380065 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:15.380178 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:15.380422 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 09:09:15.380533 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":286,"threads_started":1}
I20260806 09:09:15.382050 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:15.370431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43333 (local address 127.27.187.129:39551)
0806 09:09:15.371587 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:15.371634 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:15.373560 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:15.374382 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:15.374392 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:15.374492 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:15.375199 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:15.375219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.376325 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.376329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:15.377609 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:15.377623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.378479 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.378486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:15.378795 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:15.379608 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:15.379610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:15.379665 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:15.379750 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:15.380167 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:15.380179 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:15.380189 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:09:15.380392 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 09:09:15.380836 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":354,"threads_started":1}
I20260806 09:09:15.383934 16434 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43333
I20260806 09:09:15.384387 16434 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:15.385243 16434 heartbeater.cc:511] Master 127.27.187.190:43333 requested a full tablet report, sending...
I20260806 09:09:15.387629 16247 ts_manager.cc:194] Registered new tserver with Master: fae693640f164c3487834302c2da74b8 (127.27.187.129:43817)
I20260806 09:09:15.389256 16247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.129:39551
W20260806 09:09:15.750761 16438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:15.751170 16438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:15.751336 16438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:15.759375 16438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:15.759544 16438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:15.759711 16438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.130
I20260806 09:09:15.769994 16438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.187.130
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.130
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:15.771760 16438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:15.773896 16438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:15.794488 16444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:15.794528 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:15.796510 16446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:15.798250 16438 server_base.cc:1034] running on GCE node
I20260806 09:09:15.799348 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:15.800993 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:15.802315 16438 hybrid_clock.cc:648] HybridClock initialized: now 1786007355802210 us; error 58 us; skew 500 ppm
I20260806 09:09:15.802906 16438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:15.806532 16438 webserver.cc:460] Webserver started at http://127.27.187.130:46445/ using document root <none> and password file <none>
I20260806 09:09:15.807443 16438 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:15.807600 16438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:15.807989 16438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:15.813125 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data/instance:
uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "fbb91a7b29f5bb31beef815d69b65885"
server_key_iv: "36b4914c0a807edf6906286f1f9067d6"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.814586 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance:
uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "fbb91a7b29f5bb31beef815d69b65885"
server_key_iv: "36b4914c0a807edf6906286f1f9067d6"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.822782 16438 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260806 09:09:15.828167 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.830461 16438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260806 09:09:15.830732 16438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal
uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a"
format_stamp: "Formatted at 2026-08-06 09:09:15 on dist-test-slave-8tdl"
server_key: "fbb91a7b29f5bb31beef815d69b65885"
server_key_iv: "36b4914c0a807edf6906286f1f9067d6"
server_key_version: "encryptionkey@0"
I20260806 09:09:15.831039 16438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:15.864214 16438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:15.865336 16438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:15.865711 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:15.867162 16438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:15.869019 16438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:15.869167 16438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.869331 16438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:15.869433 16438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:15.910063 16438 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.130:43807
I20260806 09:09:15.910141 16567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.130:43807 every 8 connection(s)
I20260806 09:09:15.911700 16438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/data/info.pb
I20260806 09:09:15.916607 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16438
I20260806 09:09:15.916826 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal/instance
I20260806 09:09:15.917344 28398 external_mini_cluster.cc:1428] Setting key d193305103df911b94c5ab77439c72af
I20260806 09:09:15.920992 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.187.131:0
--local_ip_for_outbound_sockets=127.27.187.131
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--builtin_ntp_servers=127.27.187.148:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:09:15.928746 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:15.914863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.130:41163 (local address 127.27.187.190:43333)
0806 09:09:15.915467 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:15.915472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:15.917101 (+ 1629us) server_negotiation.cc:407] Connection header received
0806 09:09:15.919821 (+ 2720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:15.919827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:15.919943 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:15.920091 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:15.921620 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.922906 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.924759 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.925544 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.925707 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:15.927629 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:15.927638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:15.927646 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:15.927676 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:15.927736 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:15.927756 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:15.927864 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:15.928125 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 09:09:15.928281 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":365,"threads_started":1}
I20260806 09:09:15.929899 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:15.915603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43333 (local address 127.27.187.130:41163)
0806 09:09:15.916888 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:15.916952 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:15.919402 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:15.920392 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:15.920403 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:15.920533 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:15.921404 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:15.921434 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.923085 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.923090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:15.924568 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:15.924584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:15.925878 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:15.925885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:15.926146 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:15.927196 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:15.927199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:15.927266 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:15.927370 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:15.927877 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:15.927886 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:15.927893 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:09:15.928080 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 09:09:15.928515 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":325,"threads_started":1}
I20260806 09:09:15.931794 16568 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43333
I20260806 09:09:15.932219 16568 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:15.933298 16568 heartbeater.cc:511] Master 127.27.187.190:43333 requested a full tablet report, sending...
I20260806 09:09:15.935338 16247 ts_manager.cc:194] Registered new tserver with Master: 3793fc5ec86d4ea6a2c7b578a8037f9a (127.27.187.130:43807)
I20260806 09:09:15.936591 16247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.130:41163
W20260806 09:09:16.299089 16572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:16.299404 16572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:16.299566 16572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:16.307349 16572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:16.307471 16572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:16.307664 16572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.187.131
I20260806 09:09:16.317792 16572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.187.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.187.131
--webserver_port=0
--tserver_master_addrs=127.27.187.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.187.131
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:16.319403 16572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:16.321457 16572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:16.338023 16577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:16.338135 16578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:16.338160 16580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:16.344769 16572 server_base.cc:1034] running on GCE node
I20260806 09:09:16.345713 16572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:16.347043 16572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:16.348495 16572 hybrid_clock.cc:648] HybridClock initialized: now 1786007356348477 us; error 65 us; skew 500 ppm
I20260806 09:09:16.348906 16572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:09:16.351845 16572 webserver.cc:460] Webserver started at http://127.27.187.131:36345/ using document root <none> and password file <none>
I20260806 09:09:16.352674 16572 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:16.352837 16572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:16.353197 16572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:16.356539 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data/instance:
uuid: "298133032d1546fbaf1fad5d9ab4435c"
format_stamp: "Formatted at 2026-08-06 09:09:16 on dist-test-slave-8tdl"
server_key: "117fafa75359ae2248dc0c229f4e5ddc"
server_key_iv: "0265097dbc64d40a1ef7245d14b41352"
server_key_version: "encryptionkey@0"
I20260806 09:09:16.357506 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance:
uuid: "298133032d1546fbaf1fad5d9ab4435c"
format_stamp: "Formatted at 2026-08-06 09:09:16 on dist-test-slave-8tdl"
server_key: "117fafa75359ae2248dc0c229f4e5ddc"
server_key_iv: "0265097dbc64d40a1ef7245d14b41352"
server_key_version: "encryptionkey@0"
I20260806 09:09:16.363905 16572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:09:16.368546 16586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:16.370246 16572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:09:16.370445 16572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal
uuid: "298133032d1546fbaf1fad5d9ab4435c"
format_stamp: "Formatted at 2026-08-06 09:09:16 on dist-test-slave-8tdl"
server_key: "117fafa75359ae2248dc0c229f4e5ddc"
server_key_iv: "0265097dbc64d40a1ef7245d14b41352"
server_key_version: "encryptionkey@0"
I20260806 09:09:16.370744 16572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:16.392709 16434 heartbeater.cc:503] Master 127.27.187.190:43333 was elected leader, sending a full tablet report...
I20260806 09:09:16.394364 16572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:16.395449 16572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:16.395989 16572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:16.397176 16572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:09:16.398885 16572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:09:16.399019 16572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:16.399169 16572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:09:16.399266 16572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:16.437917 16572 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.131:42167
I20260806 09:09:16.438025 16701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.131:42167 every 8 connection(s)
I20260806 09:09:16.439533 16572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/data/info.pb
I20260806 09:09:16.440135 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16572
I20260806 09:09:16.440367 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal/instance
I20260806 09:09:16.441151 28398 external_mini_cluster.cc:1428] Setting key 3b55858d7973840862f62608b56477f6
I20260806 09:09:16.454316 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.442800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:49619 (local address 127.27.187.190:43333)
0806 09:09:16.443464 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.443471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.445139 (+ 1668us) server_negotiation.cc:407] Connection header received
0806 09:09:16.447297 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.447303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.447416 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.447539 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:16.448726 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.449505 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.451269 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.451680 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.451824 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.453332 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:16.453338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:16.453342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:16.453365 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:16.453408 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:16.453421 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:16.453524 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.453764 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.453879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":372,"threads_started":1}
I20260806 09:09:16.455685 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.443745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:43333 (local address 127.27.187.131:49619)
0806 09:09:16.444989 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.445035 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.446959 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.447815 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.447825 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.447926 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:09:16.448537 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.448556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.449682 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.449687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.451103 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.451117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.451942 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.451949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.452208 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.452973 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:09:16.452976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:09:16.453021 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:09:16.453114 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:09:16.453523 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:09:16.453537 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:09:16.453547 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.453753 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.454216 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":361,"threads_started":1}
I20260806 09:09:16.457737 16702 heartbeater.cc:347] Connected to a master server at 127.27.187.190:43333
I20260806 09:09:16.458148 16702 heartbeater.cc:464] Registering TS with master...
I20260806 09:09:16.459184 16702 heartbeater.cc:511] Master 127.27.187.190:43333 requested a full tablet report, sending...
I20260806 09:09:16.460894 16247 ts_manager.cc:194] Registered new tserver with Master: 298133032d1546fbaf1fad5d9ab4435c (127.27.187.131:42167)
I20260806 09:09:16.461884 16247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.187.131:49619
I20260806 09:09:16.470893 28398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:09:16.487275 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.480197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.27.187.190:43333)
0806 09:09:16.480397 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.480401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.480788 (+ 387us) server_negotiation.cc:407] Connection header received
0806 09:09:16.480950 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.480954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.481025 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.481126 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:16.481751 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.482492 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.484350 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.484804 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.484935 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.486188 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:16.486193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:16.486197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:16.486253 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:16.486291 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:16.486305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:16.486385 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.486600 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.486978 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:09:16.491986 16247 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:09:16.499415 16247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:09:16.542558 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.514458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.27.187.131:42167)
0806 09:09:16.528887 (+ 14429us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.528899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.529670 (+ 771us) server_negotiation.cc:407] Connection header received
0806 09:09:16.530888 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.530901 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.531187 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.531412 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:16.532831 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.535110 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.539676 (+ 4566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.541189 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.541413 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.541679 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.541966 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.542149 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14240,"thread_start_us":14058,"threads_started":1}
I20260806 09:09:16.542699 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.514108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.131:42167 (local address 127.0.0.1:49444)
0806 09:09:16.529423 (+ 15315us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.529481 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.530608 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.531787 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.531795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.531857 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:16.532522 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.532539 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.535376 (+ 2837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.535387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.539466 (+ 4079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.539478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.541426 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.541433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.541665 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.541728 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.541906 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.542134 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15015,"mutex_wait_us":7,"spinlock_wait_cycles":3328,"thread_start_us":14815,"threads_started":1}
I20260806 09:09:16.543181 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.514915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.27.187.130:43807)
0806 09:09:16.527986 (+ 13071us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.528002 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.529841 (+ 1839us) server_negotiation.cc:407] Connection header received
0806 09:09:16.530419 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.530433 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.530796 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.531141 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:16.532697 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.535224 (+ 2527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.539490 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.541265 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.541456 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.541706 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.542372 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.542562 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12767,"thread_start_us":13099,"threads_started":1}
I20260806 09:09:16.544000 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.514485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:43807 (local address 127.0.0.1:42942)
0806 09:09:16.529681 (+ 15196us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.529706 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.529957 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.531579 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.531593 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.531644 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:16.532371 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.532396 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.535485 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.535496 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.539185 (+ 3689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.539207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.541892 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.541901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.542121 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.542170 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.543218 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.543373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15010,"mutex_wait_us":42,"spinlock_wait_cycles":9472,"thread_start_us":15254,"threads_started":1}
I20260806 09:09:16.546655 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.530231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43817 (local address 127.0.0.1:52668)
0806 09:09:16.536467 (+ 6236us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.536493 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.536654 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.538140 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.538145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.538177 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:16.538750 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.538774 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.540903 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.540909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.543454 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.543472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.545858 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.545865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.546060 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.546092 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.546179 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.546318 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6078,"mutex_wait_us":1,"thread_start_us":5877,"threads_started":1}
I20260806 09:09:16.547111 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.530575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.27.187.129:43817)
0806 09:09:16.537307 (+ 6732us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.537320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.537357 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:09:16.537598 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.537607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.537917 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.538157 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:16.538948 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.540675 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.543748 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.545631 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.545867 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.546123 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.546241 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.546436 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6469,"thread_start_us":6626,"threads_started":1}
I20260806 09:09:16.547163 16634 tablet_service.cc:1467] Processing CreateTablet for tablet 660084198b634437a51f3d4fea993605 (DEFAULT_TABLE table=test-table [id=569640295cb14184a132d8029fe79ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:16.548843 16634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660084198b634437a51f3d4fea993605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:16.548930 16502 tablet_service.cc:1467] Processing CreateTablet for tablet 660084198b634437a51f3d4fea993605 (DEFAULT_TABLE table=test-table [id=569640295cb14184a132d8029fe79ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:16.550311 16502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660084198b634437a51f3d4fea993605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:16.550912 16368 tablet_service.cc:1467] Processing CreateTablet for tablet 660084198b634437a51f3d4fea993605 (DEFAULT_TABLE table=test-table [id=569640295cb14184a132d8029fe79ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:09:16.552664 16368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660084198b634437a51f3d4fea993605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:16.562875 16721 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Bootstrap starting.
I20260806 09:09:16.565157 16722 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Bootstrap starting.
I20260806 09:09:16.567926 16721 tablet_bootstrap.cc:654] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:16.568987 16722 tablet_bootstrap.cc:654] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:16.569890 16721 log.cc:824] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:16.570185 16722 log.cc:824] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:16.575986 16723 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Bootstrap starting.
I20260806 09:09:16.577581 16722 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: No bootstrap required, opened a new log
I20260806 09:09:16.577903 16722 ts_tablet_manager.cc:1397] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260806 09:09:16.580279 16723 tablet_bootstrap.cc:654] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:16.580602 16721 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: No bootstrap required, opened a new log
I20260806 09:09:16.580916 16721 ts_tablet_manager.cc:1397] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.003s
I20260806 09:09:16.581755 16723 log.cc:824] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:16.585404 16723 tablet_bootstrap.cc:492] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: No bootstrap required, opened a new log
I20260806 09:09:16.585727 16723 ts_tablet_manager.cc:1397] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 09:09:16.586750 16722 raft_consensus.cc:348] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.587267 16722 raft_consensus.cc:374] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:16.587378 16722 raft_consensus.cc:729] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298133032d1546fbaf1fad5d9ab4435c, State: Initialized, Role: FOLLOWER
I20260806 09:09:16.588101 16722 consensus_queue.cc:260] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.590299 16702 heartbeater.cc:503] Master 127.27.187.190:43333 was elected leader, sending a full tablet report...
I20260806 09:09:16.591116 16722 ts_tablet_manager.cc:1428] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260806 09:09:16.592610 16721 raft_consensus.cc:348] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.593204 16721 raft_consensus.cc:374] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:16.593323 16721 raft_consensus.cc:729] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793fc5ec86d4ea6a2c7b578a8037f9a, State: Initialized, Role: FOLLOWER
I20260806 09:09:16.594141 16721 consensus_queue.cc:260] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.595495 16568 heartbeater.cc:503] Master 127.27.187.190:43333 was elected leader, sending a full tablet report...
I20260806 09:09:16.597179 16721 ts_tablet_manager.cc:1428] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260806 09:09:16.597115 16723 raft_consensus.cc:348] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.597615 16723 raft_consensus.cc:374] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:16.597744 16723 raft_consensus.cc:729] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae693640f164c3487834302c2da74b8, State: Initialized, Role: FOLLOWER
I20260806 09:09:16.598419 16723 consensus_queue.cc:260] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.600517 16723 ts_tablet_manager.cc:1428] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260806 09:09:16.619400 16435 tablet.cc:2369] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:16.664300 16569 tablet.cc:2369] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:09:16.691854 16703 tablet.cc:2369] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:09:16.739820 16727 raft_consensus.cc:482] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:09:16.740128 16727 raft_consensus.cc:504] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.741881 16727 leader_election.cc:288] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3793fc5ec86d4ea6a2c7b578a8037f9a (127.27.187.130:43807), fae693640f164c3487834302c2da74b8 (127.27.187.129:43817)
I20260806 09:09:16.749867 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.742355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.130:43807 (local address 127.27.187.131:59259)
0806 09:09:16.742559 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.742583 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.742783 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.743620 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.743624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.743651 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:16.743853 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.743861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.745558 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.745565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.747621 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.747642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.748984 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.748990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.749210 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.749253 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.749375 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.749545 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:09:16.749938 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.742430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:59259 (local address 127.27.187.130:43807)
0806 09:09:16.742650 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.742655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.742727 (+ 72us) server_negotiation.cc:407] Connection header received
0806 09:09:16.743058 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.743066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.743218 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.743361 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:16.744036 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.745379 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.747855 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.748786 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.748927 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.748981 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.749471 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.749651 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:09:16.750335 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.742882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.129:43817 (local address 127.27.187.131:41825)
0806 09:09:16.743714 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 09:09:16.743732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:09:16.743909 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:09:16.744388 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:09:16.744394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:09:16.744430 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:09:16.744841 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.744855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.746351 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.746360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.747983 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:09:16.748000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.749521 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.749527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:09:16.749720 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.749756 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:09:16.749865 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 09:09:16.750016 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":443,"threads_started":1}
I20260806 09:09:16.750327 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.743221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.187.131:41825 (local address 127.27.187.129:43817)
0806 09:09:16.743458 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.743463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.743842 (+ 379us) server_negotiation.cc:407] Connection header received
0806 09:09:16.744093 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.744097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.744176 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.744254 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:09:16.745082 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.746141 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.748264 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.749319 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.749483 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.749547 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.749919 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.750063 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:09:16.751216 16522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660084198b634437a51f3d4fea993605" candidate_uuid: "298133032d1546fbaf1fad5d9ab4435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" is_pre_election: true
I20260806 09:09:16.751466 16389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660084198b634437a51f3d4fea993605" candidate_uuid: "298133032d1546fbaf1fad5d9ab4435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae693640f164c3487834302c2da74b8" is_pre_election: true
I20260806 09:09:16.751775 16522 raft_consensus.cc:2454] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298133032d1546fbaf1fad5d9ab4435c in term 0.
I20260806 09:09:16.752103 16389 raft_consensus.cc:2454] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298133032d1546fbaf1fad5d9ab4435c in term 0.
I20260806 09:09:16.752599 16589 leader_election.cc:302] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298133032d1546fbaf1fad5d9ab4435c, 3793fc5ec86d4ea6a2c7b578a8037f9a; no voters:
I20260806 09:09:16.753022 16727 raft_consensus.cc:2781] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:09:16.753159 16727 raft_consensus.cc:482] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:09:16.753306 16727 raft_consensus.cc:3037] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:16.755821 16727 raft_consensus.cc:504] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.756497 16727 leader_election.cc:288] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [CANDIDATE]: Term 1 election: Requested vote from peers 3793fc5ec86d4ea6a2c7b578a8037f9a (127.27.187.130:43807), fae693640f164c3487834302c2da74b8 (127.27.187.129:43817)
I20260806 09:09:16.757294 16522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660084198b634437a51f3d4fea993605" candidate_uuid: "298133032d1546fbaf1fad5d9ab4435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a"
I20260806 09:09:16.757360 16389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660084198b634437a51f3d4fea993605" candidate_uuid: "298133032d1546fbaf1fad5d9ab4435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae693640f164c3487834302c2da74b8"
I20260806 09:09:16.757572 16522 raft_consensus.cc:3037] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:16.757582 16389 raft_consensus.cc:3037] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:16.761138 16389 raft_consensus.cc:2454] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298133032d1546fbaf1fad5d9ab4435c in term 1.
I20260806 09:09:16.761137 16522 raft_consensus.cc:2454] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298133032d1546fbaf1fad5d9ab4435c in term 1.
I20260806 09:09:16.761782 16589 leader_election.cc:302] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298133032d1546fbaf1fad5d9ab4435c, 3793fc5ec86d4ea6a2c7b578a8037f9a; no voters:
I20260806 09:09:16.762156 16727 raft_consensus.cc:2781] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:16.762841 16727 raft_consensus.cc:686] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 1 LEADER]: Becoming Leader. State: Replica: 298133032d1546fbaf1fad5d9ab4435c, State: Running, Role: LEADER
I20260806 09:09:16.763479 16727 consensus_queue.cc:237] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } }
I20260806 09:09:16.768829 16247 catalog_manager.cc:5495] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c reported cstate change: term changed from 0 to 1, leader changed from <none> to 298133032d1546fbaf1fad5d9ab4435c (127.27.187.131). New cstate: current_term: 1 leader_uuid: "298133032d1546fbaf1fad5d9ab4435c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "298133032d1546fbaf1fad5d9ab4435c" member_type: VOTER last_known_addr { host: "127.27.187.131" port: 42167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 } health_report { overall_health: UNKNOWN } } }
I20260806 09:09:16.793071 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.786601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.27.187.131:42167)
0806 09:09:16.786880 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.786886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.786906 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:09:16.787031 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.787035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.787133 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.787283 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:09:16.787836 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.788624 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.790254 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.790865 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.790988 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.791730 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:09:16.792345 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:09:16.792469 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.792664 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.792743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260806 09:09:16.803150 16389 raft_consensus.cc:1261] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 1 FOLLOWER]: Refusing update from remote peer 298133032d1546fbaf1fad5d9ab4435c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:16.803149 16522 raft_consensus.cc:1261] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 1 FOLLOWER]: Refusing update from remote peer 298133032d1546fbaf1fad5d9ab4435c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:09:16.804250 16731 consensus_queue.cc:1035] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3793fc5ec86d4ea6a2c7b578a8037f9a" member_type: VOTER last_known_addr { host: "127.27.187.130" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:16.804721 16727 consensus_queue.cc:1035] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae693640f164c3487834302c2da74b8" member_type: VOTER last_known_addr { host: "127.27.187.129" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:09:16.819198 16734 mvcc.cc:205] Tried to move back new op lower bound from 7315486133453549568 to 7315486133307539456. Current Snapshot: MvccSnapshot[applied={T|T < 7315486133453549568}]
I20260806 09:09:16.820312 16735 mvcc.cc:205] Tried to move back new op lower bound from 7315486133453549568 to 7315486133307539456. Current Snapshot: MvccSnapshot[applied={T|T < 7315486133453549568}]
I20260806 09:09:16.835053 16246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:16.835474 16246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:09:16.839924 16246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0de70cfb23514e698a33012863e4ea3c: Sending DeleteTablet for 3 replicas of tablet 660084198b634437a51f3d4fea993605
I20260806 09:09:16.841483 16634 tablet_service.cc:1514] Processing DeleteTablet for tablet 660084198b634437a51f3d4fea993605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:16 UTC) from {username='slave'} at 127.0.0.1:49444
I20260806 09:09:16.841456 16368 tablet_service.cc:1514] Processing DeleteTablet for tablet 660084198b634437a51f3d4fea993605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:16 UTC) from {username='slave'} at 127.0.0.1:52668
I20260806 09:09:16.841497 16502 tablet_service.cc:1514] Processing DeleteTablet for tablet 660084198b634437a51f3d4fea993605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:09:16 UTC) from {username='slave'} at 127.0.0.1:42942
I20260806 09:09:16.844033 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16306
I20260806 09:09:16.854655 16744 tablet_replica.cc:330] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: stopping tablet replica
I20260806 09:09:16.855358 16744 raft_consensus.cc:2229] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:16.856086 16744 raft_consensus.cc:2258] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:16.857432 16746 tablet_replica.cc:330] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: stopping tablet replica
I20260806 09:09:16.858001 16746 raft_consensus.cc:2229] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:09:16.858605 16746 raft_consensus.cc:2258] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:16.859589 16744 ts_tablet_manager.cc:1905] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:16.859669 16745 tablet_replica.cc:330] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: stopping tablet replica
I20260806 09:09:16.860236 16745 raft_consensus.cc:2229] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:09:16.861027 16745 raft_consensus.cc:2258] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:09:16.863998 16745 ts_tablet_manager.cc:1905] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:16.868778 16744 ts_tablet_manager.cc:1918] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:16.869410 16744 log.cc:1196] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-0/wal/wals/660084198b634437a51f3d4fea993605
I20260806 09:09:16.870219 16744 ts_tablet_manager.cc:1939] T 660084198b634437a51f3d4fea993605 P fae693640f164c3487834302c2da74b8: Deleting consensus metadata
I20260806 09:09:16.872063 16745 ts_tablet_manager.cc:1918] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:16.872303 16745 log.cc:1196] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-2/wal/wals/660084198b634437a51f3d4fea993605
I20260806 09:09:16.872895 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:16.845916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.27.187.129:43817)
0806 09:09:16.846153 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:16.846161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:16.861269 (+ 15108us) server_negotiation.cc:407] Connection header received
0806 09:09:16.862388 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:16.862396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:16.862527 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:16.862639 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:16.863503 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.864607 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.866798 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:16.867542 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:16.867647 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:16.870912 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:16.870925 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:16.870943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:16.871781 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:16.871858 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:16.871887 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:16.871997 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:16.872306 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 09:09:16.872455 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:09:16.872893 16745 ts_tablet_manager.cc:1939] T 660084198b634437a51f3d4fea993605 P 298133032d1546fbaf1fad5d9ab4435c: Deleting consensus metadata
I20260806 09:09:16.872798 16233 catalog_manager.cc:4841] TS fae693640f164c3487834302c2da74b8 (127.27.187.129:43817): tablet 660084198b634437a51f3d4fea993605 (table test-table [id=569640295cb14184a132d8029fe79ec8]) successfully deleted
I20260806 09:09:16.873546 16746 ts_tablet_manager.cc:1905] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:09:16.874049 16429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:16.874917 16232 catalog_manager.cc:4841] TS 298133032d1546fbaf1fad5d9ab4435c (127.27.187.131:42167): tablet 660084198b634437a51f3d4fea993605 (table test-table [id=569640295cb14184a132d8029fe79ec8]) successfully deleted
I20260806 09:09:16.883033 16746 ts_tablet_manager.cc:1918] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:09:16.883275 16746 log.cc:1196] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Deleting WAL directory at /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007003447047-28398-0/minicluster-data/ts-1/wal/wals/660084198b634437a51f3d4fea993605
I20260806 09:09:16.883961 16746 ts_tablet_manager.cc:1939] T 660084198b634437a51f3d4fea993605 P 3793fc5ec86d4ea6a2c7b578a8037f9a: Deleting consensus metadata
I20260806 09:09:16.886301 16233 catalog_manager.cc:4841] TS 3793fc5ec86d4ea6a2c7b578a8037f9a (127.27.187.130:43807): tablet 660084198b634437a51f3d4fea993605 (table test-table [id=569640295cb14184a132d8029fe79ec8]) successfully deleted
I20260806 09:09:16.977675 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16306
I20260806 09:09:16.999271 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16438
I20260806 09:09:17.009097 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:17.000163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.27.187.130:43807)
0806 09:09:17.000345 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:17.000350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:17.000539 (+ 189us) server_negotiation.cc:407] Connection header received
0806 09:09:17.000770 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:17.000776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:17.000908 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:17.001012 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:17.001599 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:17.003156 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:17.005285 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:17.005738 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:17.005855 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:17.007403 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:17.007413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:17.007428 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:17.008221 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:17.008280 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:17.008304 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:17.008408 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:17.008596 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 09:09:17.008717 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:09:17.010227 16563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:17.103523 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16438
I20260806 09:09:17.128677 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16572
I20260806 09:09:17.137568 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:17.129634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.27.187.131:42167)
0806 09:09:17.129866 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:17.129873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:17.129915 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:09:17.130020 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:17.130026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:17.130131 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:17.130251 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:17.130808 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:17.132107 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:17.133543 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:17.134230 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:17.134375 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:17.135759 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:17.135770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:17.135789 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:17.136646 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:17.136720 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:17.136754 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:17.136869 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:17.137038 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 09:09:17.137154 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:09:17.138511 16697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:17.261471 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16572
I20260806 09:09:17.282958 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16216
I20260806 09:09:17.284217 16277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:17.398386 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16216
2026-08-06T09:09:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3284 ms)
[----------] 6 tests from EncryptionPolicyTest (24805 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:17 dist-test-slave-8tdl krb5kdc[16754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:17 dist-test-slave-8tdl krb5kdc[16754](info): set up 2 sockets
Aug 06 09:09:17 dist-test-slave-8tdl krb5kdc[16754](info): commencing operation
krb5kdc: starting...
W20260806 09:09:18.834765 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:19 dist-test-slave-8tdl krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:19.204046 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:19.581634 16768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:19.581966 16768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:19.582137 16768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:19.590327 16768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:19.590480 16768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:19.590581 16768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:19.590736 16768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:19.590832 16768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:19.602293 16768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:19.603797 16768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:19.605238 16768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:19.619062 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:19.619220 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:19.619244 16773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:19.620510 16768 server_base.cc:1034] running on GCE node
I20260806 09:09:19.621474 16768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:19.623359 16768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:19.624627 16768 hybrid_clock.cc:648] HybridClock initialized: now 1786007359624577 us; error 51 us; skew 500 ppm
I20260806 09:09:19.625051 16768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:19 dist-test-slave-8tdl krb5kdc[16754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:19.632932 16768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:19.634968 16768 webserver.cc:460] Webserver started at http://127.0.0.1:36681/ using document root <none> and password file <none>
I20260806 09:09:19.635818 16768 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:19.635998 16768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:19.636386 16768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:19.639999 16768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "5f2b8047381b41bc8e23b8824338dc55"
format_stamp: "Formatted at 2026-08-06 09:09:19 on dist-test-slave-8tdl"
server_key: "2fe278d94361eb9d21b5bb9d2b8a0591"
server_key_iv: "2e64c9c3e3032bb7f3ed1cb44168a078"
server_key_version: "encryptionkey@0"
I20260806 09:09:19.640952 16768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "5f2b8047381b41bc8e23b8824338dc55"
format_stamp: "Formatted at 2026-08-06 09:09:19 on dist-test-slave-8tdl"
server_key: "2fe278d94361eb9d21b5bb9d2b8a0591"
server_key_iv: "2e64c9c3e3032bb7f3ed1cb44168a078"
server_key_version: "encryptionkey@0"
I20260806 09:09:19.647491 16768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:09:19.651933 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:19.653678 16768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:09:19.653950 16768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "5f2b8047381b41bc8e23b8824338dc55"
format_stamp: "Formatted at 2026-08-06 09:09:19 on dist-test-slave-8tdl"
server_key: "2fe278d94361eb9d21b5bb9d2b8a0591"
server_key_iv: "2e64c9c3e3032bb7f3ed1cb44168a078"
server_key_version: "encryptionkey@0"
I20260806 09:09:19.654192 16768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:19.695021 16768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:19.700065 16768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:19.700414 16768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:19.723245 16768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37381
I20260806 09:09:19.723258 16835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37381 every 8 connection(s)
I20260806 09:09:19.725502 16768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:19.731405 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:19.732471 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16768
I20260806 09:09:19.732703 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:19.733417 28398 external_mini_cluster.cc:1428] Setting key 05c852f3694bc1b70b9f91b701a02fbb
I20260806 09:09:19.747954 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55: Bootstrap starting.
Aug 06 09:09:19 dist-test-slave-8tdl krb5kdc[16754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:09:19.752296 16836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:19.753569 16836 log.cc:824] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:19.757325 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55: No bootstrap required, opened a new log
I20260806 09:09:19.762934 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:19.735572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60389 (local address 127.0.0.1:37381)
0806 09:09:19.736479 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:19.736491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:19.736540 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:09:19.738280 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:19.738312 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:19.738915 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:19.739322 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:19.741342 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:19.743044 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:19.745101 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:19.746191 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:19.746372 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:19.752464 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:19.752480 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:19.752512 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:19.752582 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:19.756526 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:19.757736 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:19.757747 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:19.757758 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:19.758052 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:19.758923 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:19.758929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:19.758933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:19.759748 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:19.760128 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:19.760832 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 09:09:19.761281 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":409,"threads_started":1}
I20260806 09:09:19.767838 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:19.767985 16836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260806 09:09:19.768513 16836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:19.768623 16836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2b8047381b41bc8e23b8824338dc55, State: Initialized, Role: FOLLOWER
I20260806 09:09:19.769352 16836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260806 09:09:19.769663 16836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:19.769795 16836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:19.769958 16836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:19.773818 16836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260806 09:09:19.774590 16836 leader_election.cc:302] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f2b8047381b41bc8e23b8824338dc55; no voters:
I20260806 09:09:19.775439 16836 leader_election.cc:288] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:19.775549 16846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:19.778586 16836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:19.779102 16846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2b8047381b41bc8e23b8824338dc55, State: Running, Role: LEADER
I20260806 09:09:19.779800 16846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260806 09:09:19.783852 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2b8047381b41bc8e23b8824338dc55. Latest consensus state: current_term: 1 leader_uuid: "5f2b8047381b41bc8e23b8824338dc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260806 09:09:19.784610 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:19.785718 16847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f2b8047381b41bc8e23b8824338dc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2b8047381b41bc8e23b8824338dc55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260806 09:09:19.786341 16847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:19.794324 16856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:19.797384 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:19.774247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53545 (local address 127.0.0.1:37381)
0806 09:09:19.774471 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:19.774478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:19.775196 (+ 718us) server_negotiation.cc:407] Connection header received
0806 09:09:19.775454 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:19.775459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:19.775567 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:19.775688 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:19.777964 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:19.779124 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:19.781370 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:19.782472 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:19.782713 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:19.787820 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:19.787831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:19.787840 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:19.787911 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:19.793034 (+ 5123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:19.794363 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:19.794374 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:19.794377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:19.794660 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:19.795368 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:19.795376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:19.795386 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:19.795893 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:19.796210 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:19.796770 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 09:09:19.796995 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:09:19.800187 16856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:19.810215 16856 catalog_manager.cc:1347] Generated new cluster ID: 70cf1b3d8746401a8ec5c94d8a145133
I20260806 09:09:19.810365 16856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:19.828575 16856 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:19.830183 16856 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:19.845798 16856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f2b8047381b41bc8e23b8824338dc55: Generated new TSK 0
I20260806 09:09:19.846973 16856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:19.861672 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16768
I20260806 09:09:19.863128 16831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:19.965133 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16768
W20260806 09:09:19.988901 16752 connection.cc:535] client connection to 127.0.0.1:37381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37381): Connection reset by peer (error 104)
2026-08-06T09:09:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:20 dist-test-slave-8tdl krb5kdc[16869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:20 dist-test-slave-8tdl krb5kdc[16869](info): set up 2 sockets
Aug 06 09:09:20 dist-test-slave-8tdl krb5kdc[16869](info): commencing operation
krb5kdc: starting...
W20260806 09:09:21.174360 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:21 dist-test-slave-8tdl krb5kdc[16869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:21.539538 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:21.927944 16882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:21.928295 16882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:21.928468 16882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:21.936501 16882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:21.936645 16882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:21.936738 16882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:21.936859 16882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:21.936925 16882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:21.948361 16882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:21.949877 16882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:21.951459 16882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:21.965499 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:21.965461 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:21.965426 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:21.966466 16882 server_base.cc:1034] running on GCE node
I20260806 09:09:21.967443 16882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:21.969025 16882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:21.970297 16882 hybrid_clock.cc:648] HybridClock initialized: now 1786007361970248 us; error 47 us; skew 500 ppm
I20260806 09:09:21.970847 16882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:21 dist-test-slave-8tdl krb5kdc[16869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:21.974788 16882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:21.976801 16882 webserver.cc:460] Webserver started at http://127.0.0.1:35229/ using document root <none> and password file <none>
I20260806 09:09:21.977589 16882 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:21.977749 16882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:21.978118 16882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:21.981575 16882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "8dfa0763288549e7b1b03163c577287a"
format_stamp: "Formatted at 2026-08-06 09:09:21 on dist-test-slave-8tdl"
server_key: "59db524b224122f1705f7a2017d45578"
server_key_iv: "472e30dff98bef45b35de526914378ce"
server_key_version: "encryptionkey@0"
I20260806 09:09:21.982507 16882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "8dfa0763288549e7b1b03163c577287a"
format_stamp: "Formatted at 2026-08-06 09:09:21 on dist-test-slave-8tdl"
server_key: "59db524b224122f1705f7a2017d45578"
server_key_iv: "472e30dff98bef45b35de526914378ce"
server_key_version: "encryptionkey@0"
I20260806 09:09:21.989441 16882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:21.994059 16898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:21.995963 16882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:09:21.996228 16882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "8dfa0763288549e7b1b03163c577287a"
format_stamp: "Formatted at 2026-08-06 09:09:21 on dist-test-slave-8tdl"
server_key: "59db524b224122f1705f7a2017d45578"
server_key_iv: "472e30dff98bef45b35de526914378ce"
server_key_version: "encryptionkey@0"
I20260806 09:09:21.996491 16882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:22.021108 16882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:22.026923 16882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:22.027462 16882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:22.051287 16882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39773
I20260806 09:09:22.051301 16950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39773 every 8 connection(s)
I20260806 09:09:22.053305 16882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:22.057928 16951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:22.058012 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16882
I20260806 09:09:22.058202 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:22.058805 28398 external_mini_cluster.cc:1428] Setting key 73f17861086b08db5a75500a3dfe7f52
I20260806 09:09:22.074286 16951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a: Bootstrap starting.
I20260806 09:09:22.076582 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:22.061935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43993 (local address 127.0.0.1:39773)
0806 09:09:22.062941 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:22.062956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:22.063006 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:09:22.065262 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:22.065297 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:22.065811 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:22.066205 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:22.067039 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:22.068448 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:22.070883 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:22.071776 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:22.071937 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:22.073566 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:22.073576 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:22.073595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:22.074256 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:22.074359 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:22.074438 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:22.074630 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:22.075146 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 09:09:22.075392 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":414,"threads_started":1}
I20260806 09:09:22.078176 16951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:22.079499 16951 log.cc:824] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:22.081944 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:22.083966 16951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a: No bootstrap required, opened a new log
I20260806 09:09:22.096149 16951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } }
I20260806 09:09:22.096997 16951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:22.097148 16951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfa0763288549e7b1b03163c577287a, State: Initialized, Role: FOLLOWER
I20260806 09:09:22.097805 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:22.088835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37861 (local address 127.0.0.1:39773)
0806 09:09:22.089194 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:22.089201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:22.089605 (+ 404us) server_negotiation.cc:407] Connection header received
0806 09:09:22.090241 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:22.090249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:22.090352 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:22.090477 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:22.091174 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:22.092506 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:22.094287 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:22.095072 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:22.095239 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:22.096617 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:22.096625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:22.096630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:22.096662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:22.096730 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:22.096806 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:22.096961 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:22.097363 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 09:09:22.097432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:09:22.098259 16951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } }
I20260806 09:09:22.098628 16951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:22.098879 16951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:22.099092 16951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:22.101892 16951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } }
I20260806 09:09:22.102573 16951 leader_election.cc:302] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dfa0763288549e7b1b03163c577287a; no voters:
I20260806 09:09:22.103487 16951 leader_election.cc:288] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:22.103621 16961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:22.104454 16961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [term 1 LEADER]: Becoming Leader. State: Replica: 8dfa0763288549e7b1b03163c577287a, State: Running, Role: LEADER
I20260806 09:09:22.105129 16961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } }
I20260806 09:09:22.106009 16951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:22.108407 16962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dfa0763288549e7b1b03163c577287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } } }
I20260806 09:09:22.108911 16963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dfa0763288549e7b1b03163c577287a. Latest consensus state: current_term: 1 leader_uuid: "8dfa0763288549e7b1b03163c577287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dfa0763288549e7b1b03163c577287a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39773 } } }
I20260806 09:09:22.109197 16962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:22.109230 16963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:22.112212 16966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:22.118635 16966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:22.129667 16966 catalog_manager.cc:1347] Generated new cluster ID: 14fa53346ed547d3b9364227abe482b2
I20260806 09:09:22.129825 16966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:22.151216 16966 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:22.152904 16966 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:22.168139 16966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dfa0763288549e7b1b03163c577287a: Generated new TSK 0
I20260806 09:09:22.169356 16966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:22.189921 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16882
W20260806 09:09:22.191766 16946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43993
W20260806 09:09:22.192742 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:22.192966 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16882
W20260806 09:09:22.214795 16867 connection.cc:535] client connection to 127.0.0.1:39773 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39773): Connection reset by peer (error 104)
2026-08-06T09:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:22 dist-test-slave-8tdl krb5kdc[16982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:22 dist-test-slave-8tdl krb5kdc[16982](info): set up 2 sockets
Aug 06 09:09:22 dist-test-slave-8tdl krb5kdc[16982](info): commencing operation
krb5kdc: starting...
W20260806 09:09:23.429800 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.002s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:23 dist-test-slave-8tdl krb5kdc[16982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:23.803073 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:24.182588 16995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:24.183007 16995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:24.183230 16995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:24.191036 16995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:24.191186 16995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:24.191280 16995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:24.191376 16995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:24.191475 16995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:24.202116 16995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:24.203610 16995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:24.205103 16995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:24.219509 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:24.219524 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:24.219601 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:24.220647 16995 server_base.cc:1034] running on GCE node
I20260806 09:09:24.221638 16995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:24.223434 16995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:24.224685 16995 hybrid_clock.cc:648] HybridClock initialized: now 1786007364224622 us; error 81 us; skew 500 ppm
I20260806 09:09:24.225113 16995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:24 dist-test-slave-8tdl krb5kdc[16982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:24.228888 16995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:24.230662 16995 webserver.cc:460] Webserver started at http://127.0.0.1:36347/ using document root <none> and password file <none>
I20260806 09:09:24.231469 16995 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:24.231586 16995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:24.231855 16995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:24.235389 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "5b9096fc9e134223b7b3c2e04da8439d"
format_stamp: "Formatted at 2026-08-06 09:09:24 on dist-test-slave-8tdl"
server_key: "701c63ce90ad2c447255e50318e0b40e"
server_key_iv: "ee1ab51849adca7cb3cd83da59d85a59"
server_key_version: "encryptionkey@0"
I20260806 09:09:24.236284 16995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "5b9096fc9e134223b7b3c2e04da8439d"
format_stamp: "Formatted at 2026-08-06 09:09:24 on dist-test-slave-8tdl"
server_key: "701c63ce90ad2c447255e50318e0b40e"
server_key_iv: "ee1ab51849adca7cb3cd83da59d85a59"
server_key_version: "encryptionkey@0"
I20260806 09:09:24.242323 16995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:09:24.246372 17010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:24.247954 16995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:09:24.248193 16995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "5b9096fc9e134223b7b3c2e04da8439d"
format_stamp: "Formatted at 2026-08-06 09:09:24 on dist-test-slave-8tdl"
server_key: "701c63ce90ad2c447255e50318e0b40e"
server_key_iv: "ee1ab51849adca7cb3cd83da59d85a59"
server_key_version: "encryptionkey@0"
I20260806 09:09:24.248436 16995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:24.255494 16995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:24.260596 16995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:24.260938 16995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:24.282320 16995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37753
I20260806 09:09:24.282361 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37753 every 8 connection(s)
I20260806 09:09:24.284032 16995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:24.288307 17064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:24.290133 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 16995
I20260806 09:09:24.290352 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:24.290903 28398 external_mini_cluster.cc:1428] Setting key 5a3649e4ba87066e587fcf2932ca9e24
I20260806 09:09:24.300442 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:24.293181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46187 (local address 127.0.0.1:37753)
0806 09:09:24.294185 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:24.294200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:24.294248 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:09:24.296334 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:24.296363 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:24.296800 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:24.297290 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:24.298523 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:24.298537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:24.298557 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:24.298578 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:24.298654 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:24.298743 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:24.298949 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:24.299078 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 09:09:24.299344 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":474,"threads_started":1}
I20260806 09:09:24.302019 17064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d: Bootstrap starting.
I20260806 09:09:24.305039 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:24.305820 17064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:24.307039 17064 log.cc:824] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:24.310936 17064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d: No bootstrap required, opened a new log
I20260806 09:09:24.313086 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:24.310168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46375 (local address 127.0.0.1:37753)
0806 09:09:24.310386 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:24.310394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:24.310856 (+ 462us) server_negotiation.cc:407] Connection header received
0806 09:09:24.311064 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:24.311070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:24.311143 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:24.311256 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:24.312354 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:24.312360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:24.312365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:24.312384 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:24.312435 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:24.312451 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:24.312582 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:24.312705 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 09:09:24.312775 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:09:24.319748 17064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260806 09:09:24.320261 17064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:24.320389 17064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9096fc9e134223b7b3c2e04da8439d, State: Initialized, Role: FOLLOWER
I20260806 09:09:24.321108 17064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260806 09:09:24.321455 17064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:24.321628 17064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:24.321791 17064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:24.324570 17064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260806 09:09:24.325132 17064 leader_election.cc:302] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b9096fc9e134223b7b3c2e04da8439d; no voters:
I20260806 09:09:24.325814 17064 leader_election.cc:288] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:24.326009 17074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:24.326867 17074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [term 1 LEADER]: Becoming Leader. State: Replica: 5b9096fc9e134223b7b3c2e04da8439d, State: Running, Role: LEADER
I20260806 09:09:24.327610 17074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260806 09:09:24.328476 17064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:24.331166 17075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b9096fc9e134223b7b3c2e04da8439d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } } }
I20260806 09:09:24.331682 17075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:24.332192 17076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b9096fc9e134223b7b3c2e04da8439d. Latest consensus state: current_term: 1 leader_uuid: "5b9096fc9e134223b7b3c2e04da8439d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9096fc9e134223b7b3c2e04da8439d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } } }
I20260806 09:09:24.332428 17076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:24.335629 17083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:24.342859 17083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:24.353713 17083 catalog_manager.cc:1347] Generated new cluster ID: 5dd77191ff6242cdae90175c6add030b
I20260806 09:09:24.353901 17083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:24.366451 17083 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:24.367048 17083 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:24.374511 17083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b9096fc9e134223b7b3c2e04da8439d: Generated new TSK 0
I20260806 09:09:24.375687 17083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:24.384646 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16995
W20260806 09:09:24.386804 17058 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46187
W20260806 09:09:24.387629 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 16995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:24.387826 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 16995
2026-08-06T09:09:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:24 dist-test-slave-8tdl krb5kdc[17095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:24 dist-test-slave-8tdl krb5kdc[17095](info): set up 2 sockets
Aug 06 09:09:24 dist-test-slave-8tdl krb5kdc[17095](info): commencing operation
krb5kdc: starting...
W20260806 09:09:25.621717 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:25 dist-test-slave-8tdl krb5kdc[17095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:26.012974 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:26.391889 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:26.392367 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:26.392547 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:26.401417 17108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:26.401629 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:26.401748 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:26.401885 17108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:26.401975 17108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:26.413642 17108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:26.415199 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:26.416623 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:26.430199 17113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:26.430045 17116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:26.430277 17114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:26.430923 17108 server_base.cc:1034] running on GCE node
I20260806 09:09:26.431855 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:26.433605 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:26.434857 17108 hybrid_clock.cc:648] HybridClock initialized: now 1786007366434806 us; error 53 us; skew 500 ppm
I20260806 09:09:26.435259 17108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:26 dist-test-slave-8tdl krb5kdc[17095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:26.439188 17108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:26.441145 17108 webserver.cc:460] Webserver started at http://127.0.0.1:38487/ using document root <none> and password file <none>
I20260806 09:09:26.441974 17108 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:26.442155 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:26.442523 17108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:26.446211 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "030ab08bf06c4fabab37eb10fcd71df3"
format_stamp: "Formatted at 2026-08-06 09:09:26 on dist-test-slave-8tdl"
server_key: "c1b969f00ecaa6160945ec7b781052e6"
server_key_iv: "cf7e6282adb2b053212c5674841ca58e"
server_key_version: "encryptionkey@0"
I20260806 09:09:26.447242 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "030ab08bf06c4fabab37eb10fcd71df3"
format_stamp: "Formatted at 2026-08-06 09:09:26 on dist-test-slave-8tdl"
server_key: "c1b969f00ecaa6160945ec7b781052e6"
server_key_iv: "cf7e6282adb2b053212c5674841ca58e"
server_key_version: "encryptionkey@0"
I20260806 09:09:26.453493 17108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:09:26.457566 17123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:26.459203 17108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:26.459450 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "030ab08bf06c4fabab37eb10fcd71df3"
format_stamp: "Formatted at 2026-08-06 09:09:26 on dist-test-slave-8tdl"
server_key: "c1b969f00ecaa6160945ec7b781052e6"
server_key_iv: "cf7e6282adb2b053212c5674841ca58e"
server_key_version: "encryptionkey@0"
I20260806 09:09:26.459692 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:26.477851 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:26.483094 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:26.483479 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:26.505486 17108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45079
I20260806 09:09:26.505527 17174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45079 every 8 connection(s)
I20260806 09:09:26.507071 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:26.511013 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17108
I20260806 09:09:26.511215 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:26.511327 17175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:26.511734 28398 external_mini_cluster.cc:1428] Setting key eb9343da24e08c3c236fc651523a78cc
I20260806 09:09:26.527647 17175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3: Bootstrap starting.
Aug 06 09:09:26 dist-test-slave-8tdl krb5kdc[17095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:09:26.531291 17175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:26.532528 17175 log.cc:824] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:26.535792 17175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3: No bootstrap required, opened a new log
I20260806 09:09:26.537693 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:26.513640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44599 (local address 127.0.0.1:45079)
0806 09:09:26.514562 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:26.514574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:26.514619 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:09:26.516568 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:26.516596 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:26.517096 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:26.517472 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:26.519587 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:26.520978 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:26.522949 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:26.523766 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:26.523933 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:26.529547 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:26.529556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:26.529573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:26.529625 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:26.533112 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:26.533996 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:26.534004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:26.534011 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:26.534194 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:26.534650 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:26.534654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:26.534657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:26.535290 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:26.535625 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:26.536116 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 09:09:26.536617 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":419,"threads_started":1}
I20260806 09:09:26.541728 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:26.544399 17175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260806 09:09:26.544974 17175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:26.545104 17175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030ab08bf06c4fabab37eb10fcd71df3, State: Initialized, Role: FOLLOWER
I20260806 09:09:26.545794 17175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260806 09:09:26.546097 17175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:26.546277 17175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:26.546494 17175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:26.549086 17175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260806 09:09:26.549732 17175 leader_election.cc:302] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 030ab08bf06c4fabab37eb10fcd71df3; no voters:
I20260806 09:09:26.550491 17175 leader_election.cc:288] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:26.550606 17186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:26.553557 17175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:26.557394 17186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [term 1 LEADER]: Becoming Leader. State: Replica: 030ab08bf06c4fabab37eb10fcd71df3, State: Running, Role: LEADER
I20260806 09:09:26.558368 17186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260806 09:09:26.562107 17187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "030ab08bf06c4fabab37eb10fcd71df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } } }
I20260806 09:09:26.562634 17187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:26.564472 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:26.546395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47349 (local address 127.0.0.1:45079)
0806 09:09:26.546568 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:26.546572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:26.547148 (+ 576us) server_negotiation.cc:407] Connection header received
0806 09:09:26.547365 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:26.547371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:26.547460 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:26.547579 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:26.549136 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:26.550011 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:26.551808 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:26.552450 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:26.552611 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:26.555424 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:26.555434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:26.555443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:26.555504 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:26.559336 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:26.560142 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:26.560148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:26.560152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:26.560293 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:26.560820 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:26.560825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:26.560828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:26.561316 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:26.561601 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:26.563351 (+ 1750us) server_negotiation.cc:299] Negotiation successful
0806 09:09:26.563705 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 09:09:26.568608 17188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 030ab08bf06c4fabab37eb10fcd71df3. Latest consensus state: current_term: 1 leader_uuid: "030ab08bf06c4fabab37eb10fcd71df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030ab08bf06c4fabab37eb10fcd71df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } } }
I20260806 09:09:26.569181 17188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:26.569870 17194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:26.577292 17194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:26.588773 17194 catalog_manager.cc:1347] Generated new cluster ID: 0fc3bc2d919b4e39bedb525a89ef179a
I20260806 09:09:26.588943 17194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:26.615303 17194 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:26.617606 17194 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:26.628916 17194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 030ab08bf06c4fabab37eb10fcd71df3: Generated new TSK 0
I20260806 09:09:26.630211 17194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:26.642112 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17108
I20260806 09:09:26.644214 17170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:26.742972 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17108
2026-08-06T09:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:26 dist-test-slave-8tdl krb5kdc[17208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:26 dist-test-slave-8tdl krb5kdc[17208](info): set up 2 sockets
Aug 06 09:09:26 dist-test-slave-8tdl krb5kdc[17208](info): commencing operation
krb5kdc: starting...
W20260806 09:09:27.947511 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.009s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:28 dist-test-slave-8tdl krb5kdc[17208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:28.315244 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:28.692258 17221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:28.692602 17221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:28.692757 17221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:28.700730 17221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:28.700865 17221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:28.700973 17221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:28.701082 17221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:28.701179 17221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:28.711669 17221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:28.713176 17221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:28.714610 17221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:28.727864 17229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:28.727910 17226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:28.728152 17227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:28.729491 17221 server_base.cc:1034] running on GCE node
I20260806 09:09:28.730515 17221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:28.732338 17221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:28.733619 17221 hybrid_clock.cc:648] HybridClock initialized: now 1786007368733558 us; error 39 us; skew 500 ppm
I20260806 09:09:28.734112 17221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:28 dist-test-slave-8tdl krb5kdc[17208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:28.738170 17221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:28.740334 17221 webserver.cc:460] Webserver started at http://127.0.0.1:41813/ using document root <none> and password file <none>
I20260806 09:09:28.741163 17221 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:28.741344 17221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:28.741742 17221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:28.745500 17221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3"
format_stamp: "Formatted at 2026-08-06 09:09:28 on dist-test-slave-8tdl"
server_key: "764316d1ec23deb1271a2a68dd2342db"
server_key_iv: "e77193a78726619512640bdf2b8473a7"
server_key_version: "encryptionkey@0"
I20260806 09:09:28.746495 17221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3"
format_stamp: "Formatted at 2026-08-06 09:09:28 on dist-test-slave-8tdl"
server_key: "764316d1ec23deb1271a2a68dd2342db"
server_key_iv: "e77193a78726619512640bdf2b8473a7"
server_key_version: "encryptionkey@0"
I20260806 09:09:28.753623 17221 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:09:28.758358 17236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:28.760082 17221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:28.760262 17221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3"
format_stamp: "Formatted at 2026-08-06 09:09:28 on dist-test-slave-8tdl"
server_key: "764316d1ec23deb1271a2a68dd2342db"
server_key_iv: "e77193a78726619512640bdf2b8473a7"
server_key_version: "encryptionkey@0"
I20260806 09:09:28.760519 17221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:28.785209 17221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:28.791095 17221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:28.791414 17221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:28.813944 17221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36971
I20260806 09:09:28.813984 17289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36971 every 8 connection(s)
I20260806 09:09:28.815577 17221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:28.820192 17290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:28.822831 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17221
I20260806 09:09:28.823019 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:28.823551 28398 external_mini_cluster.cc:1428] Setting key 5c693cfbc609f49b0d300042f70968f1
I20260806 09:09:28.836122 17290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3: Bootstrap starting.
I20260806 09:09:28.838483 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:28.825718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39447 (local address 127.0.0.1:36971)
0806 09:09:28.826731 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:28.826753 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:28.827168 (+ 415us) server_negotiation.cc:407] Connection header received
0806 09:09:28.829089 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:28.829116 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:28.829640 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:28.830063 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:28.830656 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:28.832053 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:28.833541 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:28.834375 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:28.834636 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:28.835945 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:28.835953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:28.835970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:28.836637 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:28.836738 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:28.836765 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:28.836900 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:28.837149 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 09:09:28.837450 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":467,"threads_started":1}
I20260806 09:09:28.839850 17290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:28.841121 17290 log.cc:824] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:28.842908 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:28.844615 17290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3: No bootstrap required, opened a new log
I20260806 09:09:28.853525 17290 raft_consensus.cc:348] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260806 09:09:28.854027 17290 raft_consensus.cc:374] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:28.854172 17290 raft_consensus.cc:729] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71302b6612143aeaa2e6dfa4f5f4ab3, State: Initialized, Role: FOLLOWER
I20260806 09:09:28.854612 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:28.847674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39989 (local address 127.0.0.1:36971)
0806 09:09:28.847908 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:28.847917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:28.848387 (+ 470us) server_negotiation.cc:407] Connection header received
0806 09:09:28.848654 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:28.848658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:28.848732 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:28.848858 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:28.849417 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:28.850205 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:28.851688 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:28.852244 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:28.852396 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:28.853785 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:28.853790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:28.853793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:28.853815 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:28.853859 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:28.853875 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:28.853982 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:28.854145 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 09:09:28.854252 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:09:28.855022 17290 consensus_queue.cc:260] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260806 09:09:28.855355 17290 raft_consensus.cc:388] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:28.855520 17290 raft_consensus.cc:482] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:28.855721 17290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:28.858318 17290 raft_consensus.cc:504] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260806 09:09:28.858927 17290 leader_election.cc:302] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e71302b6612143aeaa2e6dfa4f5f4ab3; no voters:
I20260806 09:09:28.859622 17290 leader_election.cc:288] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:28.859786 17300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:28.860707 17300 raft_consensus.cc:686] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [term 1 LEADER]: Becoming Leader. State: Replica: e71302b6612143aeaa2e6dfa4f5f4ab3, State: Running, Role: LEADER
I20260806 09:09:28.861467 17300 consensus_queue.cc:237] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260806 09:09:28.862344 17290 sys_catalog.cc:564] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:28.864718 17302 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e71302b6612143aeaa2e6dfa4f5f4ab3. Latest consensus state: current_term: 1 leader_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } } }
I20260806 09:09:28.864601 17301 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71302b6612143aeaa2e6dfa4f5f4ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } } }
I20260806 09:09:28.865366 17301 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:28.865365 17302 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:28.869640 17307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:28.874362 17307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:28.884912 17307 catalog_manager.cc:1347] Generated new cluster ID: 07931cb1a6144c42bbf0c2550a560b8a
I20260806 09:09:28.885074 17307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:28.912674 17307 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:28.914353 17307 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:28.924860 17307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e71302b6612143aeaa2e6dfa4f5f4ab3: Generated new TSK 0
I20260806 09:09:28.926083 17307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:28.943049 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17221
W20260806 09:09:28.944563 17285 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39447
W20260806 09:09:28.945529 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17221: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:28.945704 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17221
2026-08-06T09:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:29 dist-test-slave-8tdl krb5kdc[17321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:29 dist-test-slave-8tdl krb5kdc[17321](info): set up 2 sockets
Aug 06 09:09:29 dist-test-slave-8tdl krb5kdc[17321](info): commencing operation
krb5kdc: starting...
W20260806 09:09:30.146306 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.003s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:30 dist-test-slave-8tdl krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:30.519132 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:30.898809 17334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:30.899161 17334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:30.899318 17334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:30.907156 17334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:30.907297 17334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:30.907385 17334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:30.907495 17334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:30.907608 17334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:30.918560 17334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:30.920091 17334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:30.921722 17334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:30.935926 17342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:30.935822 17339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:30.935886 17340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:30.937180 17334 server_base.cc:1034] running on GCE node
I20260806 09:09:30.938206 17334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:30.940088 17334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:30.941386 17334 hybrid_clock.cc:648] HybridClock initialized: now 1786007370941328 us; error 55 us; skew 500 ppm
I20260806 09:09:30.941864 17334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:30 dist-test-slave-8tdl krb5kdc[17321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:30.952153 17334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:30.954326 17334 webserver.cc:460] Webserver started at http://127.0.0.1:33215/ using document root <none> and password file <none>
I20260806 09:09:30.955247 17334 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:30.955428 17334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:30.955801 17334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:30.959534 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "7f9d36793b484f60b2eede3c2b9d9c5c"
format_stamp: "Formatted at 2026-08-06 09:09:30 on dist-test-slave-8tdl"
server_key: "cd906685f5986b765906bd059e33ae5a"
server_key_iv: "e55ae3f570b6a092f36bbdc101a55815"
server_key_version: "encryptionkey@0"
I20260806 09:09:30.960543 17334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "7f9d36793b484f60b2eede3c2b9d9c5c"
format_stamp: "Formatted at 2026-08-06 09:09:30 on dist-test-slave-8tdl"
server_key: "cd906685f5986b765906bd059e33ae5a"
server_key_iv: "e55ae3f570b6a092f36bbdc101a55815"
server_key_version: "encryptionkey@0"
I20260806 09:09:30.967336 17334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:09:30.971782 17349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:30.973663 17334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:09:30.973914 17334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "7f9d36793b484f60b2eede3c2b9d9c5c"
format_stamp: "Formatted at 2026-08-06 09:09:30 on dist-test-slave-8tdl"
server_key: "cd906685f5986b765906bd059e33ae5a"
server_key_iv: "e55ae3f570b6a092f36bbdc101a55815"
server_key_version: "encryptionkey@0"
I20260806 09:09:30.974184 17334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:30.981113 17334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:30.986352 17334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:30.986732 17334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:31.008814 17334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38777
I20260806 09:09:31.008929 17401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38777 every 8 connection(s)
I20260806 09:09:31.010425 17334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:31.014941 17402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:31.016028 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17334
I20260806 09:09:31.016263 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:31.016743 28398 external_mini_cluster.cc:1428] Setting key e7ba4cafdfb2415c732c972fb4198470
I20260806 09:09:31.027132 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:31.019056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48673 (local address 127.0.0.1:38777)
0806 09:09:31.020105 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:31.020122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:31.020185 (+ 63us) server_negotiation.cc:407] Connection header received
0806 09:09:31.022530 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:31.022569 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:31.023184 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:31.023743 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:31.024802 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:31.024811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:31.024830 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:31.024850 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:31.024931 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:31.024958 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:31.025197 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:31.025421 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 09:09:31.025649 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":399,"threads_started":1}
I20260806 09:09:31.031253 17402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c: Bootstrap starting.
I20260806 09:09:31.031635 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:31.035989 17402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:31.037496 17402 log.cc:824] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:31.040915 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:31.037737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57789 (local address 127.0.0.1:38777)
0806 09:09:31.037975 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:31.037985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:31.038554 (+ 569us) server_negotiation.cc:407] Connection header received
0806 09:09:31.038763 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:31.038771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:31.038848 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:31.039002 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:31.040182 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:31.040186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:31.040189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:31.040206 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:31.040248 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:31.040264 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:31.040363 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:31.040523 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 09:09:31.040574 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:09:31.041810 17402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c: No bootstrap required, opened a new log
I20260806 09:09:31.050225 17402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260806 09:09:31.050783 17402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:31.050915 17402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9d36793b484f60b2eede3c2b9d9c5c, State: Initialized, Role: FOLLOWER
I20260806 09:09:31.051611 17402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260806 09:09:31.051915 17402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:31.052103 17402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:31.052299 17402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:31.054728 17402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260806 09:09:31.055344 17402 leader_election.cc:302] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f9d36793b484f60b2eede3c2b9d9c5c; no voters:
I20260806 09:09:31.056027 17402 leader_election.cc:288] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:31.056291 17413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:31.057081 17413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [term 1 LEADER]: Becoming Leader. State: Replica: 7f9d36793b484f60b2eede3c2b9d9c5c, State: Running, Role: LEADER
I20260806 09:09:31.057798 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260806 09:09:31.058804 17402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:31.061368 17415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f9d36793b484f60b2eede3c2b9d9c5c. Latest consensus state: current_term: 1 leader_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } } }
I20260806 09:09:31.061491 17414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f9d36793b484f60b2eede3c2b9d9c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } } }
I20260806 09:09:31.061856 17415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:31.061944 17414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:31.064889 17419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:31.071261 17419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:31.082253 17419 catalog_manager.cc:1347] Generated new cluster ID: 6c2671b7c2b147e69e04abaf44ceeb21
I20260806 09:09:31.082402 17419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:31.111017 17419 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:31.111672 17419 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:31.120117 17419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f9d36793b484f60b2eede3c2b9d9c5c: Generated new TSK 0
I20260806 09:09:31.121133 17419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:31.137004 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17334
W20260806 09:09:31.138640 17397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48673
W20260806 09:09:31.139597 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17334: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:31.139772 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17334
2026-08-06T09:09:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:31 dist-test-slave-8tdl krb5kdc[17435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:31 dist-test-slave-8tdl krb5kdc[17435](info): set up 2 sockets
Aug 06 09:09:31 dist-test-slave-8tdl krb5kdc[17435](info): commencing operation
krb5kdc: starting...
W20260806 09:09:32.345790 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:32 dist-test-slave-8tdl krb5kdc[17435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:32.715670 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:33.094401 17448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:33.094780 17448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:33.094897 17448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:33.103652 17448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:33.103804 17448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:33.103857 17448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:33.103915 17448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:33.103963 17448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:33.116770 17448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:33.118346 17448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:33.120213 17448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:33.134874 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:33.134804 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:33.134946 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:33.136794 17448 server_base.cc:1034] running on GCE node
I20260806 09:09:33.137800 17448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:33.139935 17448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:33.141252 17448 hybrid_clock.cc:648] HybridClock initialized: now 1786007373141195 us; error 84 us; skew 500 ppm
I20260806 09:09:33.141778 17448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:33 dist-test-slave-8tdl krb5kdc[17435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:33.145773 17448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:33.148991 17448 webserver.cc:460] Webserver started at http://127.0.0.1:46573/ using document root <none> and password file <none>
I20260806 09:09:33.150038 17448 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:33.150213 17448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:33.150590 17448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:33.154320 17448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "ca6e1568596b459f982b43f64a516a0f"
format_stamp: "Formatted at 2026-08-06 09:09:33 on dist-test-slave-8tdl"
server_key: "1ff81db48b146a047cdb77324c324420"
server_key_iv: "a951e4b36e52d1f981a4ca651b3c29e1"
server_key_version: "encryptionkey@0"
I20260806 09:09:33.155359 17448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "ca6e1568596b459f982b43f64a516a0f"
format_stamp: "Formatted at 2026-08-06 09:09:33 on dist-test-slave-8tdl"
server_key: "1ff81db48b146a047cdb77324c324420"
server_key_iv: "a951e4b36e52d1f981a4ca651b3c29e1"
server_key_version: "encryptionkey@0"
I20260806 09:09:33.161909 17448 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:33.166247 17463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:33.167902 17448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:33.168124 17448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "ca6e1568596b459f982b43f64a516a0f"
format_stamp: "Formatted at 2026-08-06 09:09:33 on dist-test-slave-8tdl"
server_key: "1ff81db48b146a047cdb77324c324420"
server_key_iv: "a951e4b36e52d1f981a4ca651b3c29e1"
server_key_version: "encryptionkey@0"
I20260806 09:09:33.168368 17448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:33.184741 17448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:33.190373 17448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:33.190758 17448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:33.212127 17448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41983
I20260806 09:09:33.212180 17514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41983 every 8 connection(s)
I20260806 09:09:33.213711 17448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:33.218015 17515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:33.223469 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17448
I20260806 09:09:33.223708 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:33.224193 28398 external_mini_cluster.cc:1428] Setting key 35d2379ea13e402e56f15d1866186e0a
I20260806 09:09:33.232543 17515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f: Bootstrap starting.
I20260806 09:09:33.236860 17515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:33.238394 17515 log.cc:824] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:33.242519 17515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f: No bootstrap required, opened a new log
Aug 06 09:09:33 dist-test-slave-8tdl krb5kdc[17435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:09:33.252828 17515 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260806 09:09:33.253315 17515 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:33.253492 17515 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6e1568596b459f982b43f64a516a0f, State: Initialized, Role: FOLLOWER
I20260806 09:09:33.254253 17515 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260806 09:09:33.254448 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:33.226480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:48873 (local address 127.0.0.1:41983)
0806 09:09:33.227430 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:33.227446 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:33.227496 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:09:33.230410 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:33.230449 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:33.231115 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:33.231636 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:33.233491 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:33.235724 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:33.237373 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:33.238591 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:33.238861 (+ 270us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:33.244487 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:33.244496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:33.244514 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:33.244565 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:33.248310 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:33.249625 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:33.249634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:33.249642 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:33.249885 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:33.250490 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:33.250494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:33.250497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:33.251426 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:33.251760 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:33.252419 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 09:09:33.252843 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":416,"threads_started":1}
I20260806 09:09:33.254544 17515 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:33.254712 17515 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:33.254938 17515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:33.257665 17515 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260806 09:09:33.258330 17515 leader_election.cc:302] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6e1568596b459f982b43f64a516a0f; no voters:
I20260806 09:09:33.259121 17515 leader_election.cc:288] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:33.259316 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:33.259332 17520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:33.261273 17515 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:33.268844 17520 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [term 1 LEADER]: Becoming Leader. State: Replica: ca6e1568596b459f982b43f64a516a0f, State: Running, Role: LEADER
I20260806 09:09:33.270015 17520 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } }
I20260806 09:09:33.278041 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6e1568596b459f982b43f64a516a0f. Latest consensus state: current_term: 1 leader_uuid: "ca6e1568596b459f982b43f64a516a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } } }
I20260806 09:09:33.279011 17521 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6e1568596b459f982b43f64a516a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6e1568596b459f982b43f64a516a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41983 } } }
I20260806 09:09:33.279347 17521 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:33.278627 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:33.283233 17539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:33.294519 17539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:33.299806 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:33.269176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:50753 (local address 127.0.0.1:41983)
0806 09:09:33.269423 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:33.269431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:33.276050 (+ 6619us) server_negotiation.cc:407] Connection header received
0806 09:09:33.276491 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:33.276515 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:33.276631 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:33.276721 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:33.279650 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:33.281169 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:33.283404 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:33.283923 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:33.284102 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:33.289556 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:33.289569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:33.289579 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:33.289662 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:33.294889 (+ 5227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:33.296310 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:33.296317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:33.296322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:33.296525 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:33.297286 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:33.297292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:33.297296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:33.297797 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:33.298172 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:33.299041 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 09:09:33.299329 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:09:33.309366 17539 catalog_manager.cc:1347] Generated new cluster ID: e90a0c658d894d8d88db669209b17666
I20260806 09:09:33.309576 17539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:33.338325 17539 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:33.339823 17539 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:33.347405 17539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6e1568596b459f982b43f64a516a0f: Generated new TSK 0
I20260806 09:09:33.348431 17539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:33.359505 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17448
I20260806 09:09:33.361135 17510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:33.461567 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17448
W20260806 09:09:33.483300 17433 connection.cc:535] client connection to 127.0.0.1:41983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41983): Connection reset by peer (error 104)
2026-08-06T09:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:33 dist-test-slave-8tdl krb5kdc[17548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:33 dist-test-slave-8tdl krb5kdc[17548](info): set up 2 sockets
Aug 06 09:09:33 dist-test-slave-8tdl krb5kdc[17548](info): commencing operation
krb5kdc: starting...
W20260806 09:09:34.690284 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:34 dist-test-slave-8tdl krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:35.064663 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:35.440021 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:35.440442 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:35.440629 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:35.448393 17561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:35.448604 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:35.448724 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:35.448846 17561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:35.448935 17561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:35.460062 17561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:35.461561 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:35.463066 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:35.477190 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:35.477274 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:35.477387 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:35.478829 17561 server_base.cc:1034] running on GCE node
I20260806 09:09:35.479812 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:35.481568 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:35.482858 17561 hybrid_clock.cc:648] HybridClock initialized: now 1786007375482786 us; error 53 us; skew 500 ppm
I20260806 09:09:35.483403 17561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:35 dist-test-slave-8tdl krb5kdc[17548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:35.487181 17561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:35.489193 17561 webserver.cc:460] Webserver started at http://127.0.0.1:46843/ using document root <none> and password file <none>
I20260806 09:09:35.490044 17561 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:35.490247 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:35.490602 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:35.493968 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "5d7a1e6e59f94300bf58888c712b3fd4"
format_stamp: "Formatted at 2026-08-06 09:09:35 on dist-test-slave-8tdl"
server_key: "d7cdcb61e6fda45c2910e65ca8f02907"
server_key_iv: "3d8868d5d316453638022dd650448ede"
server_key_version: "encryptionkey@0"
I20260806 09:09:35.494902 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "5d7a1e6e59f94300bf58888c712b3fd4"
format_stamp: "Formatted at 2026-08-06 09:09:35 on dist-test-slave-8tdl"
server_key: "d7cdcb61e6fda45c2910e65ca8f02907"
server_key_iv: "3d8868d5d316453638022dd650448ede"
server_key_version: "encryptionkey@0"
I20260806 09:09:35.500977 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:35.505162 17577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:35.507079 17561 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:09:35.507340 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "5d7a1e6e59f94300bf58888c712b3fd4"
format_stamp: "Formatted at 2026-08-06 09:09:35 on dist-test-slave-8tdl"
server_key: "d7cdcb61e6fda45c2910e65ca8f02907"
server_key_iv: "3d8868d5d316453638022dd650448ede"
server_key_version: "encryptionkey@0"
I20260806 09:09:35.507599 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:35.542793 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:35.548619 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:35.548985 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:35.570822 17561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39057
I20260806 09:09:35.570922 17628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39057 every 8 connection(s)
I20260806 09:09:35.572427 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:35.576502 17629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:35.582861 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17561
I20260806 09:09:35.583094 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:35.583560 28398 external_mini_cluster.cc:1428] Setting key fde7e14bccd78e76033acc7682da032d
I20260806 09:09:35.589346 17629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4: Bootstrap starting.
I20260806 09:09:35.593937 17629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:35.595453 17629 log.cc:824] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:35.599435 17629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4: No bootstrap required, opened a new log
I20260806 09:09:35.600757 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:35.585923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:56079 (local address 127.0.0.1:39057)
0806 09:09:35.587059 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:35.587073 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:35.587118 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:09:35.589251 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:35.589279 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:35.589814 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:35.590206 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:35.591054 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:35.592633 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:35.595039 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:35.596043 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:35.596323 (+ 280us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:35.597812 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:35.597820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:35.597837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:35.598536 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:35.598620 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:35.598650 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:35.598881 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:35.599414 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 09:09:35.599629 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":526,"threads_started":1}
I20260806 09:09:35.605366 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:35.608460 17629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260806 09:09:35.608942 17629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:35.609087 17629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7a1e6e59f94300bf58888c712b3fd4, State: Initialized, Role: FOLLOWER
I20260806 09:09:35.609735 17629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260806 09:09:35.610021 17629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:35.610181 17629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:35.610376 17629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:35.614029 17629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260806 09:09:35.614741 17629 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7a1e6e59f94300bf58888c712b3fd4; no voters:
I20260806 09:09:35.615705 17629 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:35.615841 17639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:35.618304 17629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:35.618952 17639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d7a1e6e59f94300bf58888c712b3fd4, State: Running, Role: LEADER
I20260806 09:09:35.619812 17639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260806 09:09:35.620325 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:35.611066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59369 (local address 127.0.0.1:39057)
0806 09:09:35.611291 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:35.611301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:35.611329 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:09:35.611461 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:35.611492 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:35.611605 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:35.611742 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:35.612419 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:35.613662 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:35.615976 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:35.616867 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:35.617073 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:35.618954 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:35.618962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:35.618969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:35.619005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:35.619069 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:35.619094 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:35.619703 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:35.619891 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 09:09:35.619959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:09:35.623415 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7a1e6e59f94300bf58888c712b3fd4. Latest consensus state: current_term: 1 leader_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260806 09:09:35.623710 17640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7a1e6e59f94300bf58888c712b3fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260806 09:09:35.624001 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:35.624001 17640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:35.625092 17648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:35.632421 17648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:35.643380 17648 catalog_manager.cc:1347] Generated new cluster ID: acc3eb711a3a4a27a15448a0824881e8
I20260806 09:09:35.643519 17648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:35.660106 17648 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:35.661612 17648 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:35.673225 17648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7a1e6e59f94300bf58888c712b3fd4: Generated new TSK 0
I20260806 09:09:35.674496 17648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:35.688155 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17561
W20260806 09:09:35.689826 17624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:56079
W20260806 09:09:35.690824 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17561: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:35.691013 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17561
W20260806 09:09:35.713186 17546 connection.cc:535] client connection to 127.0.0.1:39057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39057): Connection reset by peer (error 104)
2026-08-06T09:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:35 dist-test-slave-8tdl krb5kdc[17660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:35 dist-test-slave-8tdl krb5kdc[17660](info): set up 2 sockets
Aug 06 09:09:35 dist-test-slave-8tdl krb5kdc[17660](info): commencing operation
krb5kdc: starting...
W20260806 09:09:36.889968 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:37 dist-test-slave-8tdl krb5kdc[17660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:37.268040 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:37.668179 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:37.668512 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:37.668627 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:37.676266 17673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:37.676409 17673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:37.676457 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:37.676522 17673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:37.676568 17673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:37.687258 17673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:37.688613 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:37.690061 17673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:37.704245 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:37.704401 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:37.704345 17681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:37.705709 17673 server_base.cc:1034] running on GCE node
I20260806 09:09:37.706741 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:37.708514 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:37.709836 17673 hybrid_clock.cc:648] HybridClock initialized: now 1786007377709751 us; error 80 us; skew 500 ppm
I20260806 09:09:37.710362 17673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:37 dist-test-slave-8tdl krb5kdc[17660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:37.714947 17673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:37.717453 17673 webserver.cc:460] Webserver started at http://127.0.0.1:42847/ using document root <none> and password file <none>
I20260806 09:09:37.718333 17673 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:37.718524 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:37.719070 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:37.723569 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "d8942eb3f16d4c7cb9ee55482b4b1471"
format_stamp: "Formatted at 2026-08-06 09:09:37 on dist-test-slave-8tdl"
server_key: "48af769f4f528b36fca75283278d62cb"
server_key_iv: "5f5083cd44d062d16c3fcbad631c14ed"
server_key_version: "encryptionkey@0"
I20260806 09:09:37.724990 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "d8942eb3f16d4c7cb9ee55482b4b1471"
format_stamp: "Formatted at 2026-08-06 09:09:37 on dist-test-slave-8tdl"
server_key: "48af769f4f528b36fca75283278d62cb"
server_key_iv: "5f5083cd44d062d16c3fcbad631c14ed"
server_key_version: "encryptionkey@0"
I20260806 09:09:37.734045 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260806 09:09:37.739236 17688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:37.741820 17673 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:09:37.742156 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "d8942eb3f16d4c7cb9ee55482b4b1471"
format_stamp: "Formatted at 2026-08-06 09:09:37 on dist-test-slave-8tdl"
server_key: "48af769f4f528b36fca75283278d62cb"
server_key_iv: "5f5083cd44d062d16c3fcbad631c14ed"
server_key_version: "encryptionkey@0"
I20260806 09:09:37.742465 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:37.750556 17673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:37.757000 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:37.757689 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:37.781916 17673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42565
I20260806 09:09:37.781919 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42565 every 8 connection(s)
I20260806 09:09:37.783772 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:37.788777 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:37.792752 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17673
I20260806 09:09:37.792950 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:37.793570 28398 external_mini_cluster.cc:1428] Setting key 62855cb56578a11cd68d78a90da748e1
I20260806 09:09:37.803402 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:37.796076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:35567 (local address 127.0.0.1:42565)
0806 09:09:37.797036 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:37.797057 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:37.797106 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:09:37.799050 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:37.799078 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:37.799474 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:37.799876 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:37.801241 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:37.801248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:37.801263 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:37.801289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:37.801375 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:37.801400 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:37.801646 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:37.801790 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 09:09:37.802115 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":514,"threads_started":1}
I20260806 09:09:37.804946 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471: Bootstrap starting.
I20260806 09:09:37.808004 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:37.809056 17742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:37.810559 17742 log.cc:824] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:37.814657 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471: No bootstrap required, opened a new log
I20260806 09:09:37.817744 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:37.814363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:46535 (local address 127.0.0.1:42565)
0806 09:09:37.814583 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:37.814590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:37.815173 (+ 583us) server_negotiation.cc:407] Connection header received
0806 09:09:37.815414 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:37.815419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:37.815475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:37.815610 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:37.816872 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:37.816877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:37.816881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:37.816904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:37.816955 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:37.816973 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:37.817133 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:37.817352 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 09:09:37.817432 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:09:37.825187 17742 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260806 09:09:37.825666 17742 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:37.825803 17742 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8942eb3f16d4c7cb9ee55482b4b1471, State: Initialized, Role: FOLLOWER
I20260806 09:09:37.826612 17742 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260806 09:09:37.827037 17742 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:37.827234 17742 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:37.827440 17742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:37.830029 17742 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260806 09:09:37.830633 17742 leader_election.cc:302] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8942eb3f16d4c7cb9ee55482b4b1471; no voters:
I20260806 09:09:37.831419 17742 leader_election.cc:288] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:37.831656 17752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:37.832705 17752 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [term 1 LEADER]: Becoming Leader. State: Replica: d8942eb3f16d4c7cb9ee55482b4b1471, State: Running, Role: LEADER
I20260806 09:09:37.833431 17752 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260806 09:09:37.834394 17742 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:37.837025 17754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } } }
I20260806 09:09:37.837507 17754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:37.837625 17753 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8942eb3f16d4c7cb9ee55482b4b1471. Latest consensus state: current_term: 1 leader_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8942eb3f16d4c7cb9ee55482b4b1471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } } }
I20260806 09:09:37.838577 17753 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:37.841523 17760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:37.848912 17760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:37.860244 17760 catalog_manager.cc:1347] Generated new cluster ID: f1dc1b41e51a4e089261a49e17443c5a
I20260806 09:09:37.860391 17760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:37.889875 17760 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:37.890453 17760 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:37.911222 17760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8942eb3f16d4c7cb9ee55482b4b1471: Generated new TSK 0
I20260806 09:09:37.912508 17760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:37.928153 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17673
W20260806 09:09:37.929493 17737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:35567
W20260806 09:09:37.930397 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17673: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:37.930569 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17673
2026-08-06T09:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:38 dist-test-slave-8tdl krb5kdc[17773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:38 dist-test-slave-8tdl krb5kdc[17773](info): set up 2 sockets
Aug 06 09:09:38 dist-test-slave-8tdl krb5kdc[17773](info): commencing operation
krb5kdc: starting...
W20260806 09:09:39.142486 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:39 dist-test-slave-8tdl krb5kdc[17773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:39.507594 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:39.907693 17786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:39.908108 17786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:39.908277 17786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:39.916154 17786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:39.916356 17786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:39.916478 17786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:39.916591 17786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:39.916682 17786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:39.929008 17786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:39.930569 17786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:39.932071 17786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:39.945384 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:39.945622 17791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:39.945430 17792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:39.946460 17786 server_base.cc:1034] running on GCE node
I20260806 09:09:39.947362 17786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:39.948871 17786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:39.950229 17786 hybrid_clock.cc:648] HybridClock initialized: now 1786007379950169 us; error 41 us; skew 500 ppm
I20260806 09:09:39.950649 17786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:39 dist-test-slave-8tdl krb5kdc[17773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:39.954556 17786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:39.956698 17786 webserver.cc:460] Webserver started at http://127.0.0.1:36029/ using document root <none> and password file <none>
I20260806 09:09:39.957553 17786 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:39.957756 17786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:39.958123 17786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:39.961796 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "ebd3d079473444889473ab4558b87826"
format_stamp: "Formatted at 2026-08-06 09:09:39 on dist-test-slave-8tdl"
server_key: "80e4f94706231e0aa7b4e61ffc8a5b44"
server_key_iv: "7843e6e95835c86237512426b6957d18"
server_key_version: "encryptionkey@0"
I20260806 09:09:39.962990 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "ebd3d079473444889473ab4558b87826"
format_stamp: "Formatted at 2026-08-06 09:09:39 on dist-test-slave-8tdl"
server_key: "80e4f94706231e0aa7b4e61ffc8a5b44"
server_key_iv: "7843e6e95835c86237512426b6957d18"
server_key_version: "encryptionkey@0"
I20260806 09:09:39.969465 17786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 09:09:39.973982 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:39.975818 17786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:09:39.976069 17786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "ebd3d079473444889473ab4558b87826"
format_stamp: "Formatted at 2026-08-06 09:09:39 on dist-test-slave-8tdl"
server_key: "80e4f94706231e0aa7b4e61ffc8a5b44"
server_key_iv: "7843e6e95835c86237512426b6957d18"
server_key_version: "encryptionkey@0"
I20260806 09:09:39.976332 17786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:39.997001 17786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:40.003392 17786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:40.003827 17786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:40.025471 17786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39779
I20260806 09:09:40.025528 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39779 every 8 connection(s)
I20260806 09:09:40.027072 17786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:40.031383 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:40.034938 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17786
I20260806 09:09:40.035231 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:40.035712 28398 external_mini_cluster.cc:1428] Setting key aaced36d2c0934208d9ecc35d6a0716e
I20260806 09:09:40.045468 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826: Bootstrap starting.
I20260806 09:09:40.049844 17853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:40.051430 17853 log.cc:824] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:09:40 dist-test-slave-8tdl krb5kdc[17773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:09:40.055433 17853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826: No bootstrap required, opened a new log
I20260806 09:09:40.063421 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:40.037910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:56761 (local address 127.0.0.1:39779)
0806 09:09:40.038876 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:40.038894 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:40.038942 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:09:40.040768 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:40.040795 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:40.041296 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:40.041754 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:40.043815 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:40.045242 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:40.047250 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:40.048070 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:40.048234 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:40.054131 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:40.054140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:40.054157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:40.054199 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:40.058094 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:40.059279 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:40.059286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:40.059293 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:40.059487 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:40.060008 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:40.060011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:40.060013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:40.060652 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:40.060982 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:40.061629 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 09:09:40.062147 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":407,"threads_started":1}
I20260806 09:09:40.065445 17853 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } }
I20260806 09:09:40.065881 17853 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:40.066006 17853 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3d079473444889473ab4558b87826, State: Initialized, Role: FOLLOWER
I20260806 09:09:40.066715 17853 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } }
I20260806 09:09:40.067028 17853 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:40.067198 17853 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:40.067405 17853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:40.068352 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:40.069878 17853 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } }
I20260806 09:09:40.070441 17853 leader_election.cc:302] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebd3d079473444889473ab4558b87826; no voters:
I20260806 09:09:40.071317 17853 leader_election.cc:288] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:40.071460 17859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:40.072743 17859 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [term 1 LEADER]: Becoming Leader. State: Replica: ebd3d079473444889473ab4558b87826, State: Running, Role: LEADER
I20260806 09:09:40.073477 17859 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } }
I20260806 09:09:40.074241 17853 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:40.077102 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd3d079473444889473ab4558b87826. Latest consensus state: current_term: 1 leader_uuid: "ebd3d079473444889473ab4558b87826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } } }
I20260806 09:09:40.077814 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:40.077790 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebd3d079473444889473ab4558b87826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd3d079473444889473ab4558b87826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39779 } } }
I20260806 09:09:40.080577 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:40.088562 17869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:40.097872 17869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:40.101910 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:40.073577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53279 (local address 127.0.0.1:39779)
0806 09:09:40.073757 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:40.073782 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:40.074437 (+ 655us) server_negotiation.cc:407] Connection header received
0806 09:09:40.074754 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:40.074763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:40.074855 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:40.074984 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:40.077184 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:40.078930 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:40.081910 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:40.082912 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:40.083055 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:40.088191 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:40.088205 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:40.088217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:40.088289 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:40.096523 (+ 8234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:40.097989 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:40.097997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:40.098003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:40.098174 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:40.098879 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:40.098889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:40.098893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:40.099325 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:40.099597 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:40.101258 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0806 09:09:40.101490 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:09:40.118373 17869 catalog_manager.cc:1347] Generated new cluster ID: 9c76d6fefbb44719b1d141fe8dc69f3b
I20260806 09:09:40.118544 17869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:40.143873 17869 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:40.145977 17869 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:40.153987 17869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd3d079473444889473ab4558b87826: Generated new TSK 0
I20260806 09:09:40.155138 17869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:40.172362 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17786
I20260806 09:09:40.174577 17848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:40.274485 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17786
W20260806 09:09:40.296090 17771 connection.cc:535] client connection to 127.0.0.1:39779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39779): Connection reset by peer (error 104)
2026-08-06T09:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:40 dist-test-slave-8tdl krb5kdc[17886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:40 dist-test-slave-8tdl krb5kdc[17886](info): set up 2 sockets
Aug 06 09:09:40 dist-test-slave-8tdl krb5kdc[17886](info): commencing operation
krb5kdc: starting...
W20260806 09:09:41.478479 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:41 dist-test-slave-8tdl krb5kdc[17886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:41.855409 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:42.237713 17899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:42.238131 17899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:42.238312 17899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:42.247382 17899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:42.247607 17899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:42.247737 17899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:42.247869 17899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:42.247968 17899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:42.260272 17899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:42.261890 17899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:42.263429 17899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:42.276903 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:42.277218 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:42.277791 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:42.278241 17899 server_base.cc:1034] running on GCE node
I20260806 09:09:42.279342 17899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:42.281143 17899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:42.282369 17899 hybrid_clock.cc:648] HybridClock initialized: now 1786007382282325 us; error 36 us; skew 500 ppm
I20260806 09:09:42.282831 17899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:42 dist-test-slave-8tdl krb5kdc[17886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:42.293634 17899 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:42.295627 17899 webserver.cc:460] Webserver started at http://127.0.0.1:43383/ using document root <none> and password file <none>
I20260806 09:09:42.296417 17899 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:42.296586 17899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:42.296926 17899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:42.300508 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "3d3dd902266c44278bf11c07a06ce845"
format_stamp: "Formatted at 2026-08-06 09:09:42 on dist-test-slave-8tdl"
server_key: "b862d47843128ee0b82d8bad29394007"
server_key_iv: "eede3c1e4c737e620531447492e4813c"
server_key_version: "encryptionkey@0"
I20260806 09:09:42.301394 17899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "3d3dd902266c44278bf11c07a06ce845"
format_stamp: "Formatted at 2026-08-06 09:09:42 on dist-test-slave-8tdl"
server_key: "b862d47843128ee0b82d8bad29394007"
server_key_iv: "eede3c1e4c737e620531447492e4813c"
server_key_version: "encryptionkey@0"
I20260806 09:09:42.307953 17899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:09:42.312350 17915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:42.313984 17899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:09:42.314211 17899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "3d3dd902266c44278bf11c07a06ce845"
format_stamp: "Formatted at 2026-08-06 09:09:42 on dist-test-slave-8tdl"
server_key: "b862d47843128ee0b82d8bad29394007"
server_key_iv: "eede3c1e4c737e620531447492e4813c"
server_key_version: "encryptionkey@0"
I20260806 09:09:42.314483 17899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:42.334020 17899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:42.339020 17899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:42.339370 17899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:42.361524 17899 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33261
I20260806 09:09:42.361552 17966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33261 every 8 connection(s)
I20260806 09:09:42.363178 17899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:42.364751 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 17899
I20260806 09:09:42.364966 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:42.365559 28398 external_mini_cluster.cc:1428] Setting key 9248fe526938a4ca9207a18703136a2d
I20260806 09:09:42.368492 17967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:42.379631 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:42.367693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:47413 (local address 127.0.0.1:33261)
0806 09:09:42.368513 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:42.368525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:42.368599 (+ 74us) server_negotiation.cc:407] Connection header received
0806 09:09:42.370331 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:42.370355 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:42.370862 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:42.371306 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:42.371888 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:42.373268 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:42.374670 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:42.375425 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:42.375630 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:42.377086 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:42.377094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:42.377109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:42.377728 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:42.377823 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:42.377859 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:42.378050 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:42.378361 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 09:09:42.378541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260806 09:09:42.380638 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845: Bootstrap starting.
I20260806 09:09:42.383852 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:42.384225 17967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:42.385303 17967 log.cc:824] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:42.388576 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845: No bootstrap required, opened a new log
I20260806 09:09:42.396347 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:42.388584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:41403 (local address 127.0.0.1:33261)
0806 09:09:42.388791 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:42.388797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:42.389328 (+ 531us) server_negotiation.cc:407] Connection header received
0806 09:09:42.389496 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:42.389502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:42.389590 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:42.389707 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:42.390299 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:42.391532 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:42.393179 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:42.393946 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:42.394101 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:42.395500 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:42.395506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:42.395512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:42.395541 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:42.395595 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:42.395616 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:42.395739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:42.395929 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 09:09:42.395994 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:09:42.398836 17967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260806 09:09:42.399228 17967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:42.399315 17967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3dd902266c44278bf11c07a06ce845, State: Initialized, Role: FOLLOWER
I20260806 09:09:42.399937 17967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260806 09:09:42.400154 17967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:42.400260 17967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:42.400436 17967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:42.403208 17967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260806 09:09:42.403775 17967 leader_election.cc:302] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3dd902266c44278bf11c07a06ce845; no voters:
I20260806 09:09:42.404450 17967 leader_election.cc:288] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:42.404634 17978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:42.405493 17978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3dd902266c44278bf11c07a06ce845, State: Running, Role: LEADER
I20260806 09:09:42.406240 17978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } }
I20260806 09:09:42.407127 17967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:42.409528 17980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3dd902266c44278bf11c07a06ce845. Latest consensus state: current_term: 1 leader_uuid: "3d3dd902266c44278bf11c07a06ce845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } } }
I20260806 09:09:42.409946 17980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:42.410605 17979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3dd902266c44278bf11c07a06ce845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3dd902266c44278bf11c07a06ce845" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33261 } } }
I20260806 09:09:42.411453 17979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:42.413107 17986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:42.418828 17986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:42.430922 17986 catalog_manager.cc:1347] Generated new cluster ID: 754a546ce56d4a1ebab2bff367a24336
I20260806 09:09:42.431100 17986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:42.485939 17986 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:42.488063 17986 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:42.517807 17986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d3dd902266c44278bf11c07a06ce845: Generated new TSK 0
I20260806 09:09:42.519114 17986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:42.530458 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17899
W20260806 09:09:42.532312 17962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:47413
W20260806 09:09:42.533183 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 17899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:42.533355 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 17899
W20260806 09:09:42.554091 17884 connection.cc:535] client connection to 127.0.0.1:33261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33261): Connection reset by peer (error 104)
2026-08-06T09:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:42 dist-test-slave-8tdl krb5kdc[17999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:42 dist-test-slave-8tdl krb5kdc[17999](info): set up 2 sockets
Aug 06 09:09:42 dist-test-slave-8tdl krb5kdc[17999](info): commencing operation
krb5kdc: starting...
W20260806 09:09:43.750145 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:43 dist-test-slave-8tdl krb5kdc[17999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:09:44.136845 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:44.509857 18012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:44.510202 18012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:44.510380 18012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:44.518193 18012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:44.518332 18012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:44.518426 18012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:44.518550 18012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:44.518618 18012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:44.529184 18012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:44.530710 18012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:44.532162 18012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:44.545387 18018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:44.545454 18017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:44.545419 18020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:44.546486 18012 server_base.cc:1034] running on GCE node
I20260806 09:09:44.547492 18012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:44.549161 18012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:44.550433 18012 hybrid_clock.cc:648] HybridClock initialized: now 1786007384550379 us; error 54 us; skew 500 ppm
I20260806 09:09:44.550895 18012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:44 dist-test-slave-8tdl krb5kdc[17999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:44.555152 18012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:09:44.557339 18012 webserver.cc:460] Webserver started at http://127.0.0.1:34733/ using document root <none> and password file <none>
I20260806 09:09:44.558133 18012 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:44.558295 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:44.558646 18012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:44.562009 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "60e16602e1f444ae999061deb1963e3c"
format_stamp: "Formatted at 2026-08-06 09:09:44 on dist-test-slave-8tdl"
server_key: "ccfdad9027c43d65897edd925b3094b8"
server_key_iv: "99139b64629f8bb01564231170601555"
server_key_version: "encryptionkey@0"
I20260806 09:09:44.563103 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "60e16602e1f444ae999061deb1963e3c"
format_stamp: "Formatted at 2026-08-06 09:09:44 on dist-test-slave-8tdl"
server_key: "ccfdad9027c43d65897edd925b3094b8"
server_key_iv: "99139b64629f8bb01564231170601555"
server_key_version: "encryptionkey@0"
I20260806 09:09:44.569409 18012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260806 09:09:44.573853 18027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:44.575570 18012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:09:44.575783 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "60e16602e1f444ae999061deb1963e3c"
format_stamp: "Formatted at 2026-08-06 09:09:44 on dist-test-slave-8tdl"
server_key: "ccfdad9027c43d65897edd925b3094b8"
server_key_iv: "99139b64629f8bb01564231170601555"
server_key_version: "encryptionkey@0"
I20260806 09:09:44.576017 18012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:44.582553 18012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:44.587709 18012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:44.588069 18012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:44.616137 18012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46297
I20260806 09:09:44.616180 18078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46297 every 8 connection(s)
I20260806 09:09:44.617693 18012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:44.621871 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:44.625209 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18012
I20260806 09:09:44.625523 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:44.625988 28398 external_mini_cluster.cc:1428] Setting key e6d787ba0dee174fa354f7b8711abe92
I20260806 09:09:44.635921 18079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c: Bootstrap starting.
I20260806 09:09:44.638288 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:44.627895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:54935 (local address 127.0.0.1:46297)
0806 09:09:44.631390 (+ 3495us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:44.631411 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:44.631778 (+ 367us) server_negotiation.cc:407] Connection header received
0806 09:09:44.634346 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:44.634373 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:44.634862 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:44.635315 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:44.636535 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:44.636542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:44.636557 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:44.636584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:44.636656 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:44.636679 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:44.636927 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:44.637076 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 09:09:44.637290 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3214,"thread_start_us":2954,"threads_started":1}
I20260806 09:09:44.639617 18079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:44.641113 18079 log.cc:824] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:44.642942 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:44.644995 18079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c: No bootstrap required, opened a new log
I20260806 09:09:44.651461 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:44.648559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:41925 (local address 127.0.0.1:46297)
0806 09:09:44.648784 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:44.648791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:44.649231 (+ 440us) server_negotiation.cc:407] Connection header received
0806 09:09:44.649404 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:44.649410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:44.649486 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:44.649638 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:44.650672 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:44.650711 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:44.650717 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:44.650739 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:44.650788 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:44.650815 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:44.650958 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:44.651040 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 09:09:44.651119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:09:44.655038 18079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260806 09:09:44.655557 18079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:44.655695 18079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e16602e1f444ae999061deb1963e3c, State: Initialized, Role: FOLLOWER
I20260806 09:09:44.656394 18079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260806 09:09:44.656661 18079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:44.656769 18079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:44.656896 18079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:44.659600 18079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260806 09:09:44.660326 18079 leader_election.cc:302] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e16602e1f444ae999061deb1963e3c; no voters:
I20260806 09:09:44.661060 18079 leader_election.cc:288] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:44.661226 18089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:44.662964 18089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [term 1 LEADER]: Becoming Leader. State: Replica: 60e16602e1f444ae999061deb1963e3c, State: Running, Role: LEADER
I20260806 09:09:44.663396 18079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:44.663616 18089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260806 09:09:44.666891 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e16602e1f444ae999061deb1963e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } } }
I20260806 09:09:44.667299 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:44.668263 18099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:44.668100 18092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e16602e1f444ae999061deb1963e3c. Latest consensus state: current_term: 1 leader_uuid: "60e16602e1f444ae999061deb1963e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e16602e1f444ae999061deb1963e3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } } }
I20260806 09:09:44.668514 18092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:44.674824 18099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:44.684890 18099 catalog_manager.cc:1347] Generated new cluster ID: 63b13efee1cc46e6a43ae32b47683ae6
I20260806 09:09:44.685040 18099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:44.703562 18099 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:44.704159 18099 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:44.713296 18099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e16602e1f444ae999061deb1963e3c: Generated new TSK 0
I20260806 09:09:44.714392 18099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:44.730011 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18012
W20260806 09:09:44.731805 18074 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:54935
W20260806 09:09:44.732770 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:44.732956 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18012
2026-08-06T09:09:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:44 dist-test-slave-8tdl krb5kdc[18111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:44 dist-test-slave-8tdl krb5kdc[18111](info): set up 2 sockets
Aug 06 09:09:44 dist-test-slave-8tdl krb5kdc[18111](info): commencing operation
krb5kdc: starting...
W20260806 09:09:45.919114 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.002s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:46 dist-test-slave-8tdl krb5kdc[18111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:46.303660 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:39257
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:39257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:46.680258 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:46.680588 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:46.680684 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:46.689304 18125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:46.689456 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:46.689512 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:46.689584 18125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:46.689632 18125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:46.700680 18125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:39257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:39257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:46.702023 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:46.703430 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:46.716450 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:46.716498 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:46.716571 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:46.718034 18125 server_base.cc:1034] running on GCE node
I20260806 09:09:46.719183 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:46.721020 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:46.722316 18125 hybrid_clock.cc:648] HybridClock initialized: now 1786007386722245 us; error 66 us; skew 500 ppm
I20260806 09:09:46.722841 18125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:46 dist-test-slave-8tdl krb5kdc[18111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007386, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:46.726357 18125 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:46.728412 18125 webserver.cc:460] Webserver started at http://127.27.187.190:45087/ using document root <none> and password file <none>
I20260806 09:09:46.729194 18125 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:46.729354 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:46.729694 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:46.733068 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "c8fc635e4f8e45678bf95a4535aa86a0"
format_stamp: "Formatted at 2026-08-06 09:09:46 on dist-test-slave-8tdl"
server_key: "be098699e89d16bcfbc498ae3b135e57"
server_key_iv: "9f631a1e186ef267ef2ceb11394e84ba"
server_key_version: "encryptionkey@0"
I20260806 09:09:46.733983 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "c8fc635e4f8e45678bf95a4535aa86a0"
format_stamp: "Formatted at 2026-08-06 09:09:46 on dist-test-slave-8tdl"
server_key: "be098699e89d16bcfbc498ae3b135e57"
server_key_iv: "9f631a1e186ef267ef2ceb11394e84ba"
server_key_version: "encryptionkey@0"
I20260806 09:09:46.740446 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 09:09:46.744851 18141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:46.746518 18125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:09:46.746784 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "c8fc635e4f8e45678bf95a4535aa86a0"
format_stamp: "Formatted at 2026-08-06 09:09:46 on dist-test-slave-8tdl"
server_key: "be098699e89d16bcfbc498ae3b135e57"
server_key_iv: "9f631a1e186ef267ef2ceb11394e84ba"
server_key_version: "encryptionkey@0"
I20260806 09:09:46.747044 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:46.768158 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:46.773728 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:46.774077 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:46.795493 18125 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:39257
I20260806 09:09:46.795524 18193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:39257 every 8 connection(s)
I20260806 09:09:46.797079 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:46.800521 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18125
I20260806 09:09:46.800753 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:46.801221 28398 external_mini_cluster.cc:1428] Setting key 9423acb3c2b73c96d1eeb2841139747d
I20260806 09:09:46.802021 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:09:46 dist-test-slave-8tdl krb5kdc[18111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:46.816077 18194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0: Bootstrap starting.
I20260806 09:09:46.820094 18194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:46.821368 18194 log.cc:824] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:46.824620 18194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0: No bootstrap required, opened a new log
I20260806 09:09:46.827342 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:46.803455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59495 (local address 127.27.187.190:39257)
0806 09:09:46.804353 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:46.804365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:46.804408 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:09:46.806083 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:46.806108 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:46.806567 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:46.807081 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:46.808744 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:46.810158 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:46.811611 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:46.812372 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:46.812636 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:46.817772 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:46.817783 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:46.817799 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:46.817852 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:46.821919 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:46.823228 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:46.823237 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:46.823244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:46.823450 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:46.824031 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:46.824035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:46.824038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:46.824711 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:46.825142 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:46.825701 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 09:09:46.826083 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":454,"threads_started":1}
I20260806 09:09:46.833046 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:46.835549 18194 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } }
I20260806 09:09:46.836159 18194 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:46.836270 18194 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fc635e4f8e45678bf95a4535aa86a0, State: Initialized, Role: FOLLOWER
I20260806 09:09:46.837188 18194 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } }
I20260806 09:09:46.837589 18194 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:46.837795 18194 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:46.838001 18194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:46.841759 18194 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } }
I20260806 09:09:46.842518 18194 leader_election.cc:302] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8fc635e4f8e45678bf95a4535aa86a0; no voters:
I20260806 09:09:46.843371 18194 leader_election.cc:288] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:46.843545 18204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:46.845680 18194 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:46.850318 18204 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [term 1 LEADER]: Becoming Leader. State: Replica: c8fc635e4f8e45678bf95a4535aa86a0, State: Running, Role: LEADER
I20260806 09:09:46.851294 18204 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } }
I20260806 09:09:46.855156 18206 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fc635e4f8e45678bf95a4535aa86a0. Latest consensus state: current_term: 1 leader_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } } }
I20260806 09:09:46.855654 18206 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:46.855666 18205 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fc635e4f8e45678bf95a4535aa86a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 39257 } } }
I20260806 09:09:46.856413 18205 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:46.865406 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:46.839269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50121 (local address 127.27.187.190:39257)
0806 09:09:46.839561 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:46.839572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:46.839992 (+ 420us) server_negotiation.cc:407] Connection header received
0806 09:09:46.840183 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:46.840188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:46.840260 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:46.840355 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:46.842485 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:46.843916 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:46.846994 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:46.848160 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:46.849421 (+ 1261us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:46.852729 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:46.852737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:46.852745 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:46.852796 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:46.859078 (+ 6282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:46.860743 (+ 1665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:46.860749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:46.860754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:46.861104 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:46.861937 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:46.861946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:46.861952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:46.862488 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:46.864290 (+ 1802us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:46.864540 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 09:09:46.864817 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:09:46.868361 18217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:46.873610 18217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:46.883244 18217 catalog_manager.cc:1347] Generated new cluster ID: 5ecd40037bb04ed99e1b5df24657860e
I20260806 09:09:46.883394 18217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:46.914182 18217 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:46.915828 18217 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:46.932813 18217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8fc635e4f8e45678bf95a4535aa86a0: Generated new TSK 0
I20260806 09:09:46.934120 18217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:46.953791 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18125
I20260806 09:09:46.955302 18188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:47.058499 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18125
W20260806 09:09:47.080103 18109 connection.cc:535] client connection to 127.27.187.190:39257 recv error: Network error: failed to read from TLS socket (remote: 127.27.187.190:39257): Connection reset by peer (error 104)
2026-08-06T09:09:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:47 dist-test-slave-8tdl krb5kdc[18226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:47 dist-test-slave-8tdl krb5kdc[18226](info): set up 2 sockets
Aug 06 09:09:47 dist-test-slave-8tdl krb5kdc[18226](info): commencing operation
krb5kdc: starting...
W20260806 09:09:48.270298 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:48 dist-test-slave-8tdl krb5kdc[18226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:48.643196 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:37345
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:37345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:49.030408 18239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:49.030833 18239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:49.031009 18239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:49.038928 18239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:49.039074 18239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:49.039124 18239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:49.039211 18239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:49.039291 18239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:49.051805 18239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:37345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:37345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:49.053371 18239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:49.054903 18239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:49.068310 18247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:49.068508 18245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:49.069123 18244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:49.069525 18239 server_base.cc:1034] running on GCE node
I20260806 09:09:49.070398 18239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:49.072149 18239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:49.073417 18239 hybrid_clock.cc:648] HybridClock initialized: now 1786007389073371 us; error 36 us; skew 500 ppm
I20260806 09:09:49.073899 18239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:49 dist-test-slave-8tdl krb5kdc[18226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007389, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:49.083499 18239 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:49.085502 18239 webserver.cc:460] Webserver started at http://127.27.187.190:34789/ using document root <none> and password file <none>
I20260806 09:09:49.086383 18239 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:49.086562 18239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:49.086980 18239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:49.091171 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "e60d71e31e484ac2994e7ab8c964c6aa"
format_stamp: "Formatted at 2026-08-06 09:09:49 on dist-test-slave-8tdl"
server_key: "afd313e5a1d11545abc84fd95f0338a4"
server_key_iv: "b6b4f4ad250b64a02ff888a83bf7a989"
server_key_version: "encryptionkey@0"
I20260806 09:09:49.092254 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "e60d71e31e484ac2994e7ab8c964c6aa"
format_stamp: "Formatted at 2026-08-06 09:09:49 on dist-test-slave-8tdl"
server_key: "afd313e5a1d11545abc84fd95f0338a4"
server_key_iv: "b6b4f4ad250b64a02ff888a83bf7a989"
server_key_version: "encryptionkey@0"
I20260806 09:09:49.099110 18239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 09:09:49.103966 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:49.105924 18239 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:09:49.106175 18239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "e60d71e31e484ac2994e7ab8c964c6aa"
format_stamp: "Formatted at 2026-08-06 09:09:49 on dist-test-slave-8tdl"
server_key: "afd313e5a1d11545abc84fd95f0338a4"
server_key_iv: "b6b4f4ad250b64a02ff888a83bf7a989"
server_key_version: "encryptionkey@0"
I20260806 09:09:49.106421 18239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:49.133912 18239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:49.139042 18239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:49.139403 18239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:49.161477 18239 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:37345
I20260806 09:09:49.161543 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:37345 every 8 connection(s)
I20260806 09:09:49.163085 18239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:49.167631 18307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:49.172874 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18239
I20260806 09:09:49.173132 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:49.173815 28398 external_mini_cluster.cc:1428] Setting key 85f939cf8bfb3f6f81e265f37529128e
I20260806 09:09:49.183701 18307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa: Bootstrap starting.
I20260806 09:09:49.187934 18307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:49.189544 18307 log.cc:824] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:49.190066 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:49.176181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42503 (local address 127.27.187.190:37345)
0806 09:09:49.177107 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:49.177121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:49.177169 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:09:49.179016 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:49.179045 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:49.179555 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:49.179962 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:49.180697 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:49.182359 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:49.184440 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:49.185310 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:49.185498 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:49.187112 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:49.187126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:49.187144 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:49.187799 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:49.187904 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:49.187939 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:49.188177 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:49.188521 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 09:09:49.188724 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":441,"threads_started":1}
I20260806 09:09:49.193413 18307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa: No bootstrap required, opened a new log
I20260806 09:09:49.195175 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:49.204568 18307 raft_consensus.cc:348] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } }
I20260806 09:09:49.205281 18307 raft_consensus.cc:374] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:49.205443 18307 raft_consensus.cc:729] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60d71e31e484ac2994e7ab8c964c6aa, State: Initialized, Role: FOLLOWER
I20260806 09:09:49.206272 18307 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } }
I20260806 09:09:49.206641 18307 raft_consensus.cc:388] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:49.206857 18307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:49.207094 18307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:49.208606 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:49.200095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49165 (local address 127.27.187.190:37345)
0806 09:09:49.200335 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:49.200342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:49.200842 (+ 500us) server_negotiation.cc:407] Connection header received
0806 09:09:49.201043 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:49.201048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:49.201129 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:49.201245 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:49.201999 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:49.202916 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:49.205153 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:49.205984 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:49.206138 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:49.207750 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:49.207759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:49.207762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:49.207786 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:49.207855 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:49.207874 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:49.208023 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:49.208239 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 09:09:49.208315 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:09:49.210438 18307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } }
I20260806 09:09:49.211202 18307 leader_election.cc:302] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e60d71e31e484ac2994e7ab8c964c6aa; no voters:
I20260806 09:09:49.211896 18307 leader_election.cc:288] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:49.212106 18317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:49.212879 18317 raft_consensus.cc:686] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [term 1 LEADER]: Becoming Leader. State: Replica: e60d71e31e484ac2994e7ab8c964c6aa, State: Running, Role: LEADER
I20260806 09:09:49.213443 18317 consensus_queue.cc:237] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } }
I20260806 09:09:49.214645 18307 sys_catalog.cc:564] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:49.216729 18319 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e60d71e31e484ac2994e7ab8c964c6aa. Latest consensus state: current_term: 1 leader_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } } }
I20260806 09:09:49.217281 18319 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:49.216843 18318 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d71e31e484ac2994e7ab8c964c6aa" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 37345 } } }
I20260806 09:09:49.219705 18318 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:49.223976 18325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:49.230218 18325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:49.241623 18325 catalog_manager.cc:1347] Generated new cluster ID: 84a620a5ce2d405294fd35c9f7b8525d
I20260806 09:09:49.241813 18325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:49.255589 18325 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:49.257170 18325 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:49.266758 18325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e60d71e31e484ac2994e7ab8c964c6aa: Generated new TSK 0
I20260806 09:09:49.268016 18325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:49.278100 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18239
W20260806 09:09:49.279788 18302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42503
W20260806 09:09:49.280790 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:49.280997 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18239
W20260806 09:09:49.301774 18224 connection.cc:535] client connection to 127.27.187.190:37345 recv error: Network error: failed to read from TLS socket (remote: 127.27.187.190:37345): Connection reset by peer (error 104)
2026-08-06T09:09:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:49 dist-test-slave-8tdl krb5kdc[18338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:49 dist-test-slave-8tdl krb5kdc[18338](info): set up 2 sockets
Aug 06 09:09:49 dist-test-slave-8tdl krb5kdc[18338](info): commencing operation
krb5kdc: starting...
W20260806 09:09:50.538302 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.005s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:50 dist-test-slave-8tdl krb5kdc[18338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:50.918532 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43797
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:51.314286 18351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:51.314640 18351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:51.314848 18351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:51.322839 18351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:51.322974 18351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:51.323024 18351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:51.323094 18351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:51.323141 18351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:51.334116 18351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:43797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:51.335614 18351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:51.337018 18351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:51.350778 18357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:51.350767 18358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:51.350752 18360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:51.351558 18351 server_base.cc:1034] running on GCE node
I20260806 09:09:51.352499 18351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:51.354202 18351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:51.355499 18351 hybrid_clock.cc:648] HybridClock initialized: now 1786007391355446 us; error 45 us; skew 500 ppm
I20260806 09:09:51.355933 18351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:51 dist-test-slave-8tdl krb5kdc[18338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007391, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:51.359668 18351 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:51.361449 18351 webserver.cc:460] Webserver started at http://127.27.187.190:43789/ using document root <none> and password file <none>
I20260806 09:09:51.362387 18351 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:51.362600 18351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:51.363036 18351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:51.366536 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "e113b3cf53324e799102c27b9e918d82"
format_stamp: "Formatted at 2026-08-06 09:09:51 on dist-test-slave-8tdl"
server_key: "711e39f07d0e2d263709bf54cb5e3cbb"
server_key_iv: "a0b9219628280302fe46c504e23f3850"
server_key_version: "encryptionkey@0"
I20260806 09:09:51.367550 18351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "e113b3cf53324e799102c27b9e918d82"
format_stamp: "Formatted at 2026-08-06 09:09:51 on dist-test-slave-8tdl"
server_key: "711e39f07d0e2d263709bf54cb5e3cbb"
server_key_iv: "a0b9219628280302fe46c504e23f3850"
server_key_version: "encryptionkey@0"
I20260806 09:09:51.373575 18351 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 09:09:51.377687 18367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:51.379570 18351 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:09:51.379812 18351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "e113b3cf53324e799102c27b9e918d82"
format_stamp: "Formatted at 2026-08-06 09:09:51 on dist-test-slave-8tdl"
server_key: "711e39f07d0e2d263709bf54cb5e3cbb"
server_key_iv: "a0b9219628280302fe46c504e23f3850"
server_key_version: "encryptionkey@0"
I20260806 09:09:51.380062 18351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:51.386761 18351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:51.391822 18351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:51.392185 18351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:51.413218 18351 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:43797
I20260806 09:09:51.413266 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:43797 every 8 connection(s)
I20260806 09:09:51.414845 18351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:51.418519 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18351
I20260806 09:09:51.418758 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:51.419332 28398 external_mini_cluster.cc:1428] Setting key 5b3413da5724070c1d23957ee1741691
I20260806 09:09:51.419385 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:51.429282 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:51.421313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50587 (local address 127.27.187.190:43797)
0806 09:09:51.422301 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:51.422313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:51.422363 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:09:51.424891 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:51.424923 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:51.425375 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:51.425820 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:51.427329 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:51.427336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:51.427352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:51.427370 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:51.427453 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:51.427480 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:51.427736 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:51.427857 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 09:09:51.428149 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":416,"threads_started":1}
I20260806 09:09:51.434273 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:51.435436 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82: Bootstrap starting.
I20260806 09:09:51.440364 18420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:51.441823 18420 log.cc:824] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:51.443728 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:51.440372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38143 (local address 127.27.187.190:43797)
0806 09:09:51.440591 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:51.440599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:51.441252 (+ 653us) server_negotiation.cc:407] Connection header received
0806 09:09:51.441442 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:51.441448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:51.441526 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:51.441679 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:51.442991 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:51.442997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:51.443000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:51.443019 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:51.443064 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:51.443080 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:51.443233 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:51.443365 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 09:09:51.443436 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:09:51.445721 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82: No bootstrap required, opened a new log
I20260806 09:09:51.455435 18420 raft_consensus.cc:348] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } }
I20260806 09:09:51.455915 18420 raft_consensus.cc:374] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:51.456050 18420 raft_consensus.cc:729] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e113b3cf53324e799102c27b9e918d82, State: Initialized, Role: FOLLOWER
I20260806 09:09:51.456804 18420 consensus_queue.cc:260] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } }
I20260806 09:09:51.457114 18420 raft_consensus.cc:388] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:51.457312 18420 raft_consensus.cc:482] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:51.457499 18420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:51.459993 18420 raft_consensus.cc:504] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } }
I20260806 09:09:51.460542 18420 leader_election.cc:302] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e113b3cf53324e799102c27b9e918d82; no voters:
I20260806 09:09:51.461154 18420 leader_election.cc:288] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:51.461357 18430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:51.462141 18430 raft_consensus.cc:686] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [term 1 LEADER]: Becoming Leader. State: Replica: e113b3cf53324e799102c27b9e918d82, State: Running, Role: LEADER
I20260806 09:09:51.462817 18430 consensus_queue.cc:237] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } }
I20260806 09:09:51.463539 18420 sys_catalog.cc:564] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:51.465862 18432 sys_catalog.cc:455] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e113b3cf53324e799102c27b9e918d82. Latest consensus state: current_term: 1 leader_uuid: "e113b3cf53324e799102c27b9e918d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } } }
I20260806 09:09:51.466048 18431 sys_catalog.cc:455] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e113b3cf53324e799102c27b9e918d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e113b3cf53324e799102c27b9e918d82" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 43797 } } }
I20260806 09:09:51.466387 18432 sys_catalog.cc:458] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:51.466385 18431 sys_catalog.cc:458] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:51.470500 18436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:51.476058 18436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:51.485977 18436 catalog_manager.cc:1347] Generated new cluster ID: eb2e4db3710f409488bd9fa9d11057c9
I20260806 09:09:51.486174 18436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:51.507093 18436 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:51.507706 18436 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:51.520790 18436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e113b3cf53324e799102c27b9e918d82: Generated new TSK 0
I20260806 09:09:51.521919 18436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:51.536079 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18351
W20260806 09:09:51.537436 18415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50587
W20260806 09:09:51.538259 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:51.538430 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18351
2026-08-06T09:09:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:51 dist-test-slave-8tdl krb5kdc[18451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:51 dist-test-slave-8tdl krb5kdc[18451](info): set up 2 sockets
Aug 06 09:09:51 dist-test-slave-8tdl krb5kdc[18451](info): commencing operation
krb5kdc: starting...
W20260806 09:09:52.778398 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:53 dist-test-slave-8tdl krb5kdc[18451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:53.163709 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33779
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:53.540921 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:53.541254 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:53.541450 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:53.549124 18464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:53.549273 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:53.549374 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:53.549468 18464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:53.549533 18464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:53.560925 18464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:53.562508 18464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:53.564127 18464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:53.578028 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:53.578135 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:53.578245 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:53.579792 18464 server_base.cc:1034] running on GCE node
I20260806 09:09:53.580801 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:53.582512 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:53.583817 18464 hybrid_clock.cc:648] HybridClock initialized: now 1786007393583762 us; error 70 us; skew 500 ppm
I20260806 09:09:53.584283 18464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:53 dist-test-slave-8tdl krb5kdc[18451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:53.588109 18464 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:53.590178 18464 webserver.cc:460] Webserver started at http://127.27.187.190:37691/ using document root <none> and password file <none>
I20260806 09:09:53.591055 18464 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:53.591218 18464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:53.591560 18464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:53.595093 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "f5c272702afa4cea89c6756b9bdb06c9"
format_stamp: "Formatted at 2026-08-06 09:09:53 on dist-test-slave-8tdl"
server_key: "38dd9ae9951b3057f05d950a6f950dbd"
server_key_iv: "d46a7b4f2e069950e7684f54cf73c8d0"
server_key_version: "encryptionkey@0"
I20260806 09:09:53.595985 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "f5c272702afa4cea89c6756b9bdb06c9"
format_stamp: "Formatted at 2026-08-06 09:09:53 on dist-test-slave-8tdl"
server_key: "38dd9ae9951b3057f05d950a6f950dbd"
server_key_iv: "d46a7b4f2e069950e7684f54cf73c8d0"
server_key_version: "encryptionkey@0"
I20260806 09:09:53.602376 18464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260806 09:09:53.607220 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:53.608938 18464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:09:53.609165 18464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "f5c272702afa4cea89c6756b9bdb06c9"
format_stamp: "Formatted at 2026-08-06 09:09:53 on dist-test-slave-8tdl"
server_key: "38dd9ae9951b3057f05d950a6f950dbd"
server_key_iv: "d46a7b4f2e069950e7684f54cf73c8d0"
server_key_version: "encryptionkey@0"
I20260806 09:09:53.609428 18464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:53.634735 18464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:53.640005 18464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:53.640355 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:53.662130 18464 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:33779
I20260806 09:09:53.662223 18530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:33779 every 8 connection(s)
I20260806 09:09:53.663722 18464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:53.668087 18531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:53.671630 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18464
I20260806 09:09:53.671844 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:53.672339 28398 external_mini_cluster.cc:1428] Setting key 12f7b0c3bf311a7dda77bf2045bf2797
I20260806 09:09:53.683158 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9: Bootstrap starting.
I20260806 09:09:53.687193 18531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:53.688359 18531 log.cc:824] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:09:53 dist-test-slave-8tdl krb5kdc[18451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:09:53.692039 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9: No bootstrap required, opened a new log
I20260806 09:09:53.700575 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:53.674731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49263 (local address 127.27.187.190:33779)
0806 09:09:53.675708 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:53.675725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:53.675802 (+ 77us) server_negotiation.cc:407] Connection header received
0806 09:09:53.678420 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:53.678461 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:53.679181 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:53.679721 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:53.681315 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:53.683457 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:53.685025 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:53.686167 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:53.686385 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:53.691972 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:53.691983 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:53.691999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:53.692063 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:53.695667 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:53.696646 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:53.696653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:53.696660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:53.696861 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:53.697300 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:53.697307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:53.697316 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:53.697934 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:53.698290 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:53.698863 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 09:09:53.699464 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":456,"threads_started":1}
I20260806 09:09:53.700718 18531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } }
I20260806 09:09:53.701243 18531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:53.701373 18531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c272702afa4cea89c6756b9bdb06c9, State: Initialized, Role: FOLLOWER
I20260806 09:09:53.702085 18531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } }
I20260806 09:09:53.702384 18531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:53.702551 18531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:53.702777 18531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:53.705575 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:53.705950 18531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } }
I20260806 09:09:53.706581 18531 leader_election.cc:302] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5c272702afa4cea89c6756b9bdb06c9; no voters:
I20260806 09:09:53.707322 18531 leader_election.cc:288] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:53.707522 18536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:53.708981 18536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [term 1 LEADER]: Becoming Leader. State: Replica: f5c272702afa4cea89c6756b9bdb06c9, State: Running, Role: LEADER
I20260806 09:09:53.709844 18536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } }
I20260806 09:09:53.710163 18531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:53.716318 18537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5c272702afa4cea89c6756b9bdb06c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } } }
I20260806 09:09:53.716823 18537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:53.717437 18538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5c272702afa4cea89c6756b9bdb06c9. Latest consensus state: current_term: 1 leader_uuid: "f5c272702afa4cea89c6756b9bdb06c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5c272702afa4cea89c6756b9bdb06c9" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33779 } } }
I20260806 09:09:53.717779 18538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:53.744985 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:53.712561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52653 (local address 127.27.187.190:33779)
0806 09:09:53.712968 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:53.712977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:53.713008 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:09:53.713127 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:53.713135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:53.713248 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:53.713405 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:53.716856 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:53.718444 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:53.721821 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:53.722922 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:53.723077 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:53.728063 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:53.728073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:53.728078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:09:53.728150 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:53.739161 (+ 11011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:53.740739 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:53.740750 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:53.740756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:53.741020 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:09:53.741925 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:09:53.741934 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:09:53.741941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:09:53.742744 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:53.743103 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:53.744023 (+ 920us) server_negotiation.cc:299] Negotiation successful
0806 09:09:53.744403 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260806 09:09:53.755698 18547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:53.763809 18547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:53.775295 18547 catalog_manager.cc:1347] Generated new cluster ID: fec055c66d6f4c31bde0b0f01c6fa2fc
I20260806 09:09:53.775449 18547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:53.788717 18547 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:53.790210 18547 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:53.796514 18547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5c272702afa4cea89c6756b9bdb06c9: Generated new TSK 0
I20260806 09:09:53.797498 18547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:53.811540 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18464
I20260806 09:09:53.813151 18526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:09:53.913902 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18464
2026-08-06T09:09:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:54 dist-test-slave-8tdl krb5kdc[18565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:54 dist-test-slave-8tdl krb5kdc[18565](info): set up 2 sockets
Aug 06 09:09:54 dist-test-slave-8tdl krb5kdc[18565](info): commencing operation
krb5kdc: starting...
W20260806 09:09:55.131497 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.002s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:55 dist-test-slave-8tdl krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:55.520457 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34005
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:55.903842 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:55.904173 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:55.904335 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:55.912089 18578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:55.912235 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:55.912283 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:55.912420 18578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:55.912533 18578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:55.923240 18578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:55.925208 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:55.926946 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:55.942713 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:55.942713 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:55.942799 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:55.949268 18578 server_base.cc:1034] running on GCE node
I20260806 09:09:55.950455 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:55.952207 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:55.953461 18578 hybrid_clock.cc:648] HybridClock initialized: now 1786007395953414 us; error 45 us; skew 500 ppm
I20260806 09:09:55.953905 18578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:55 dist-test-slave-8tdl krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007395, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:55.963323 18578 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:55.965359 18578 webserver.cc:460] Webserver started at http://127.27.187.190:38087/ using document root <none> and password file <none>
I20260806 09:09:55.966249 18578 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:55.966411 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:55.966930 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:55.970407 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "257695d5ca3e42fd807e03e1d356edac"
format_stamp: "Formatted at 2026-08-06 09:09:55 on dist-test-slave-8tdl"
server_key: "34204333b570313930aa83c690010683"
server_key_iv: "c6b6993fb570c4fea1cac09b1cb096af"
server_key_version: "encryptionkey@0"
I20260806 09:09:55.971400 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "257695d5ca3e42fd807e03e1d356edac"
format_stamp: "Formatted at 2026-08-06 09:09:55 on dist-test-slave-8tdl"
server_key: "34204333b570313930aa83c690010683"
server_key_iv: "c6b6993fb570c4fea1cac09b1cb096af"
server_key_version: "encryptionkey@0"
I20260806 09:09:55.977708 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:55.982210 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:55.984037 18578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:09:55.984272 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "257695d5ca3e42fd807e03e1d356edac"
format_stamp: "Formatted at 2026-08-06 09:09:55 on dist-test-slave-8tdl"
server_key: "34204333b570313930aa83c690010683"
server_key_iv: "c6b6993fb570c4fea1cac09b1cb096af"
server_key_version: "encryptionkey@0"
I20260806 09:09:55.984483 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:56.000463 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:56.008136 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:56.008512 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:56.035990 18578 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:34005
I20260806 09:09:56.035995 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:34005 every 8 connection(s)
I20260806 09:09:56.038051 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:56.042964 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18578
I20260806 09:09:56.043192 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:56.043915 28398 external_mini_cluster.cc:1428] Setting key 1e0a69199f5a1b131a80a9ecba2b2ca9
I20260806 09:09:56.044245 18645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:56.065258 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:56.046217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50545 (local address 127.27.187.190:34005)
0806 09:09:56.047299 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:56.047318 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:56.047380 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:09:56.051514 (+ 4134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:56.051576 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:56.052478 (+ 902us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:56.053140 (+ 662us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:56.053815 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:56.056436 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:56.058922 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:56.060428 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:56.060691 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:56.062457 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:56.062466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:56.062488 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:56.063239 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:56.063347 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:56.063378 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:56.063533 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:56.063795 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 09:09:56.064149 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":454,"threads_started":1}
I20260806 09:09:56.066577 18645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac: Bootstrap starting.
I20260806 09:09:56.070194 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:56.071798 18645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:56.073246 18645 log.cc:824] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:56.076685 18645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac: No bootstrap required, opened a new log
I20260806 09:09:56.084820 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:56.076736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47001 (local address 127.27.187.190:34005)
0806 09:09:56.077036 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:56.077047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:56.077721 (+ 674us) server_negotiation.cc:407] Connection header received
0806 09:09:56.077891 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:56.077896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:56.077994 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:56.078125 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:56.078622 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:56.079966 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:56.081584 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:09:56.082365 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:09:56.082511 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:09:56.083858 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:56.083865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:56.083870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:56.083896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:56.083955 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:56.083977 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:56.084080 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:56.084318 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 09:09:56.084420 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 09:09:56.087015 18645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } }
I20260806 09:09:56.087481 18645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:56.087601 18645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257695d5ca3e42fd807e03e1d356edac, State: Initialized, Role: FOLLOWER
I20260806 09:09:56.088284 18645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } }
I20260806 09:09:56.088510 18645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:56.088682 18645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:56.088862 18645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:56.091388 18645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } }
I20260806 09:09:56.092134 18645 leader_election.cc:302] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257695d5ca3e42fd807e03e1d356edac; no voters:
I20260806 09:09:56.092855 18645 leader_election.cc:288] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:56.092964 18655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:56.093879 18655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [term 1 LEADER]: Becoming Leader. State: Replica: 257695d5ca3e42fd807e03e1d356edac, State: Running, Role: LEADER
I20260806 09:09:56.094465 18655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } }
I20260806 09:09:56.095398 18645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:56.097828 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257695d5ca3e42fd807e03e1d356edac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } } }
I20260806 09:09:56.098256 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:56.098594 18657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257695d5ca3e42fd807e03e1d356edac. Latest consensus state: current_term: 1 leader_uuid: "257695d5ca3e42fd807e03e1d356edac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257695d5ca3e42fd807e03e1d356edac" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 34005 } } }
I20260806 09:09:56.098987 18657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:56.104033 18661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:56.110143 18661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:56.121785 18661 catalog_manager.cc:1347] Generated new cluster ID: c68c0934ac144a99982e81d9786e859e
I20260806 09:09:56.122053 18661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:56.153040 18661 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:56.155454 18661 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:56.169948 18661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257695d5ca3e42fd807e03e1d356edac: Generated new TSK 0
I20260806 09:09:56.171267 18661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:56.191962 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18578
W20260806 09:09:56.193563 18640 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50545
W20260806 09:09:56.194576 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:56.194828 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18578
2026-08-06T09:09:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:56 dist-test-slave-8tdl krb5kdc[18677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:56 dist-test-slave-8tdl krb5kdc[18677](info): set up 2 sockets
Aug 06 09:09:56 dist-test-slave-8tdl krb5kdc[18677](info): commencing operation
krb5kdc: starting...
W20260806 09:09:57.410188 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:57 dist-test-slave-8tdl krb5kdc[18677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:57.795293 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33421
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:09:58.169664 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:09:58.170007 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:09:58.170177 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:09:58.178020 18690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:09:58.178161 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:09:58.178246 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:09:58.178361 18690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:09:58.178438 18690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:09:58.189374 18690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:33421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:09:58.190873 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:09:58.192318 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:09:58.205991 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:58.205945 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:09:58.206017 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:09:58.206821 18690 server_base.cc:1034] running on GCE node
I20260806 09:09:58.207800 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:09:58.210095 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:09:58.211462 18690 hybrid_clock.cc:648] HybridClock initialized: now 1786007398211489 us; error 110 us; skew 500 ppm
I20260806 09:09:58.211910 18690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:09:58 dist-test-slave-8tdl krb5kdc[18677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007398, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:09:58.215579 18690 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:09:58.217422 18690 webserver.cc:460] Webserver started at http://127.27.187.190:45277/ using document root <none> and password file <none>
I20260806 09:09:58.218205 18690 fs_manager.cc:359] Metadata directory not provided
I20260806 09:09:58.218365 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:09:58.218757 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:09:58.222123 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "153ba49ced6a40f090969b81cb460171"
format_stamp: "Formatted at 2026-08-06 09:09:58 on dist-test-slave-8tdl"
server_key: "ddac705c9c48d1f9f020cb34107ea912"
server_key_iv: "e3cfda331bea59d01e3a1153e7dc0006"
server_key_version: "encryptionkey@0"
I20260806 09:09:58.223142 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "153ba49ced6a40f090969b81cb460171"
format_stamp: "Formatted at 2026-08-06 09:09:58 on dist-test-slave-8tdl"
server_key: "ddac705c9c48d1f9f020cb34107ea912"
server_key_iv: "e3cfda331bea59d01e3a1153e7dc0006"
server_key_version: "encryptionkey@0"
I20260806 09:09:58.229358 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:09:58.233736 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:09:58.235710 18690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:09:58.235947 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "153ba49ced6a40f090969b81cb460171"
format_stamp: "Formatted at 2026-08-06 09:09:58 on dist-test-slave-8tdl"
server_key: "ddac705c9c48d1f9f020cb34107ea912"
server_key_iv: "e3cfda331bea59d01e3a1153e7dc0006"
server_key_version: "encryptionkey@0"
I20260806 09:09:58.236222 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:09:58.243059 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:09:58.248723 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:09:58.249109 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:09:58.271198 18690 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:33421
I20260806 09:09:58.271248 18757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:33421 every 8 connection(s)
I20260806 09:09:58.272873 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:09:58.277196 18758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:09:58.283322 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18690
I20260806 09:09:58.283553 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:09:58.283982 28398 external_mini_cluster.cc:1428] Setting key f7865a76b662fbd3da0ae11e3a548338
I20260806 09:09:58.289355 18758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171: Bootstrap starting.
I20260806 09:09:58.292694 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:58.286004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58045 (local address 127.27.187.190:33421)
0806 09:09:58.287020 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:58.287034 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:58.287081 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:09:58.288868 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:58.288895 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:58.289272 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:58.289706 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:58.290927 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:58.290934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:58.290949 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:58.290965 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:58.291034 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:58.291060 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:58.291328 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:58.291486 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 09:09:58.291707 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":466,"threads_started":1}
I20260806 09:09:58.293309 18758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171: Neither blocks nor log segments found. Creating new log.
I20260806 09:09:58.294546 18758 log.cc:824] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171: Log is configured to *not* fsync() on all Append() calls
I20260806 09:09:58.296833 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:09:58.297966 18758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171: No bootstrap required, opened a new log
I20260806 09:09:58.304544 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:09:58.301650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51505 (local address 127.27.187.190:33421)
0806 09:09:58.301870 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:09:58.301879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:09:58.302247 (+ 368us) server_negotiation.cc:407] Connection header received
0806 09:09:58.302450 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:09:58.302459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:09:58.302519 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:09:58.302624 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:09:58.303777 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:09:58.303784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:09:58.303789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:09:58.303819 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:09:58.303868 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:09:58.303885 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:09:58.304060 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:09:58.304208 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 09:09:58.304282 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:09:58.307723 18758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } }
I20260806 09:09:58.308173 18758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:09:58.308291 18758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153ba49ced6a40f090969b81cb460171, State: Initialized, Role: FOLLOWER
I20260806 09:09:58.309122 18758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } }
I20260806 09:09:58.309442 18758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:09:58.309603 18758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:09:58.309830 18758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:09:58.312495 18758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } }
I20260806 09:09:58.313115 18758 leader_election.cc:302] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 153ba49ced6a40f090969b81cb460171; no voters:
I20260806 09:09:58.313907 18758 leader_election.cc:288] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:09:58.314078 18768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:09:58.314853 18768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [term 1 LEADER]: Becoming Leader. State: Replica: 153ba49ced6a40f090969b81cb460171, State: Running, Role: LEADER
I20260806 09:09:58.315577 18768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } }
I20260806 09:09:58.316347 18758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:09:58.319146 18769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "153ba49ced6a40f090969b81cb460171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } } }
I20260806 09:09:58.319273 18770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153ba49ced6a40f090969b81cb460171. Latest consensus state: current_term: 1 leader_uuid: "153ba49ced6a40f090969b81cb460171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153ba49ced6a40f090969b81cb460171" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33421 } } }
I20260806 09:09:58.319723 18770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:58.319715 18769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171 [sys.catalog]: This master's current role is: LEADER
I20260806 09:09:58.323453 18774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:09:58.329102 18774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:09:58.338744 18774 catalog_manager.cc:1347] Generated new cluster ID: 90062405f1ac49579a526879e2dc976f
I20260806 09:09:58.338891 18774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:09:58.363056 18774 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:09:58.363643 18774 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:09:58.372550 18774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 153ba49ced6a40f090969b81cb460171: Generated new TSK 0
I20260806 09:09:58.373912 18774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:09:58.390360 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18690
W20260806 09:09:58.392081 18753 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58045
W20260806 09:09:58.392964 28398 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18690: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:09:58.393173 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18690
2026-08-06T09:09:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2198 ms)
[----------] 18 tests from AuthTokenIssuingTest (40989 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:09:58 dist-test-slave-8tdl krb5kdc[18789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:09:58 dist-test-slave-8tdl krb5kdc[18789](info): set up 2 sockets
Aug 06 09:09:58 dist-test-slave-8tdl krb5kdc[18789](info): commencing operation
krb5kdc: starting...
W20260806 09:09:59.574239 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:09:59 dist-test-slave-8tdl krb5kdc[18789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:09:59.960412 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33959
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:00.345243 18802 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:00.345623 18802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:00.345691 18802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:00.345777 18802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:00.353513 18802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:00.353636 18802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:00.353704 18802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:00.353755 18802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:00.353824 18802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:00.364681 18802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:33959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:00.366071 18802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:00.367470 18802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:00.380451 18807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:00.380540 18810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:00.380568 18808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:00.381558 18802 server_base.cc:1034] running on GCE node
I20260806 09:10:00.382429 18802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:00.384022 18802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:00.385316 18802 hybrid_clock.cc:648] HybridClock initialized: now 1786007400385232 us; error 56 us; skew 500 ppm
I20260806 09:10:00.385706 18802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:00 dist-test-slave-8tdl krb5kdc[18789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:00.389310 18802 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:10:00.391131 18802 webserver.cc:460] Webserver started at http://127.27.187.190:46175/ using document root <none> and password file <none>
I20260806 09:10:00.391893 18802 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:00.391992 18802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:00.392320 18802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:00.395752 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "35f41a63d4d144ab998877d6b23c64a0"
format_stamp: "Formatted at 2026-08-06 09:10:00 on dist-test-slave-8tdl"
server_key: "fdc36e4b189657c90e581630f4ec68a0"
server_key_iv: "2a5bf921828b8e9917b1267f9329f5ab"
server_key_version: "encryptionkey@0"
I20260806 09:10:00.396675 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "35f41a63d4d144ab998877d6b23c64a0"
format_stamp: "Formatted at 2026-08-06 09:10:00 on dist-test-slave-8tdl"
server_key: "fdc36e4b189657c90e581630f4ec68a0"
server_key_iv: "2a5bf921828b8e9917b1267f9329f5ab"
server_key_version: "encryptionkey@0"
I20260806 09:10:00.402805 18802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:10:00.407097 18817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:00.408965 18802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:10:00.409185 18802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "35f41a63d4d144ab998877d6b23c64a0"
format_stamp: "Formatted at 2026-08-06 09:10:00 on dist-test-slave-8tdl"
server_key: "fdc36e4b189657c90e581630f4ec68a0"
server_key_iv: "2a5bf921828b8e9917b1267f9329f5ab"
server_key_version: "encryptionkey@0"
I20260806 09:10:00.409425 18802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:00.443666 18802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:00.449051 18802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:00.449337 18802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:00.471637 18802 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:33959
I20260806 09:10:00.471760 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:33959 every 8 connection(s)
I20260806 09:10:00.473284 18802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:00.477861 18869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:00.482808 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18802
I20260806 09:10:00.483038 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:10:00.483770 28398 external_mini_cluster.cc:1428] Setting key d7e9446132bc7de324723c1adec6428a
I20260806 09:10:00.485991 18869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:00 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:00.513512 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.489691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:33959 (local address 127.0.0.1:52592)
0806 09:10:00.490987 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.491034 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:00.493026 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:00.494016 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:00.494023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:00.494897 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:00.495288 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:00.495309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:00.496976 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:00.496982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:00.498136 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:00.498150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:00.499689 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:00.499695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:00.499842 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:00.500670 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:00.500708 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:00.503839 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:00.508758 (+ 4919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:00.508769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:00.508781 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:00.509233 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:00.509808 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:00.509812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:00.509815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:00.509986 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:00.511321 (+ 1335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:00.511327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:00.511563 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 09:10:00.511803 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 09:10:00.512337 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":790,"threads_started":1}
I20260806 09:10:00.513525 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.489681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.27.187.190:33959)
0806 09:10:00.490629 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:00.490653 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:00.491200 (+ 547us) server_negotiation.cc:407] Connection header received
0806 09:10:00.493474 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:00.493480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:00.493758 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:00.493992 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:00.495537 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:00.496796 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:00.498297 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:00.499403 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:00.499660 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:00.504156 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:00.504172 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:00.504196 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:00.504249 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:00.508551 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:00.509404 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:00.509416 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:00.509422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:00.509656 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:00.510226 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:00.510231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:00.510235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:00.511118 (+ 883us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:00.511353 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:00.512039 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 09:10:00.512560 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":656,"thread_start_us":398,"threads_started":1}
I20260806 09:10:00.517154 18869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:00.518539 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.518038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43200)
0806 09:10:00.518269 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.518403 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260806 09:10:00.518883 18821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
I20260806 09:10:00.520697 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.520290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43214)
0806 09:10:00.520436 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.520520 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:10:00.521152 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:45115: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.187.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
I20260806 09:10:00.570595 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } attempt: 1
I20260806 09:10:00.572044 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:45115
--webserver_interface=127.27.187.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:10:00.572177 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.571538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43230)
0806 09:10:00.571819 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.571953 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 09:10:00.573529 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.572994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43234)
0806 09:10:00.573212 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.573330 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 09:10:00.573848 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:45115: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
I20260806 09:10:00.642182 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } attempt: 2
I20260806 09:10:00.643610 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.643000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43250)
0806 09:10:00.643294 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.643414 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:10:00.644902 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.644509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43260)
0806 09:10:00.644677 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.644766 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 09:10:00.645205 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:45115: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
I20260806 09:10:00.736508 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } attempt: 3
I20260806 09:10:00.737983 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.737442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43262)
0806 09:10:00.737722 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.737828 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 09:10:00.739365 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.738891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43272)
0806 09:10:00.739112 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.739200 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 09:10:00.739670 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:45115: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
I20260806 09:10:00.882967 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } attempt: 4
I20260806 09:10:00.884470 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.883860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43278)
0806 09:10:00.884229 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.884323 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260806 09:10:00.885639 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:00.885245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43286)
0806 09:10:00.885402 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:00.885499 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:10:00.885908 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:45115: Network error: Client connection negotiation failed: client connection to 127.27.187.189:45115: connect: Connection refused (error 111)
W20260806 09:10:00.948058 18876 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:00.948460 18876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:00.948594 18876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:00.948755 18876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:00.956521 18876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:00.956635 18876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:00.956739 18876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:00.956820 18876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:00.956914 18876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:00.967599 18876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.187.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:00.969036 18876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:00.970453 18876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:00.984181 18884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:00.984261 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:00.984300 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:00.985351 18876 server_base.cc:1034] running on GCE node
I20260806 09:10:00.986380 18876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:00.988312 18876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:00.989637 18876 hybrid_clock.cc:648] HybridClock initialized: now 1786007400989550 us; error 68 us; skew 500 ppm
I20260806 09:10:00.990161 18876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:00 dist-test-slave-8tdl krb5kdc[18789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:00.999521 18876 init.cc:377] Logged in from keytab as kudu/127.27.187.189@KRBTEST.COM (short username kudu)
I20260806 09:10:01.001497 18876 webserver.cc:460] Webserver started at http://127.27.187.189:37891/ using document root <none> and password file <none>
I20260806 09:10:01.002353 18876 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:01.002509 18876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:01.002952 18876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:01.006392 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data/instance:
uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "7bfd8b98d231f2bf27f9fbc697aab4eb"
server_key_iv: "ff04981fd555ee066195f9fda0cb1798"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.007488 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal/instance:
uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "7bfd8b98d231f2bf27f9fbc697aab4eb"
server_key_iv: "ff04981fd555ee066195f9fda0cb1798"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.014364 18876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:10:01.018960 18891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:01.021169 18876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:10:01.021413 18876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "7bfd8b98d231f2bf27f9fbc697aab4eb"
server_key_iv: "ff04981fd555ee066195f9fda0cb1798"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.021679 18876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:01.040772 18876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:01.046132 18876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:01.046464 18876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:01.067823 18876 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.189:45115
I20260806 09:10:01.067864 18943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.189:45115 every 8 connection(s)
I20260806 09:10:01.069427 18876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
I20260806 09:10:01.070442 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18876
I20260806 09:10:01.070699 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-1/wal/instance
I20260806 09:10:01.071180 28398 external_mini_cluster.cc:1428] Setting key 51d7a1b2f81bd8950dd3d1ecbd809ec1
I20260806 09:10:01.073992 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:01.081328 18944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.187.188@KRBTEST.COM; defaulting to no policy
I20260806 09:10:01.112243 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.084485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.27.187.190:33959)
0806 09:10:01.085104 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.085118 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.086351 (+ 1233us) server_negotiation.cc:407] Connection header received
0806 09:10:01.090271 (+ 3920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.090275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.090360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.090482 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.093510 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.094475 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.097760 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.098306 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.098448 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.104362 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.104369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.104372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.104429 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.107803 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.109306 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.109311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.109314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.109472 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.110094 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.110098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.110100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.110451 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.110643 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.111530 (+ 887us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.111806 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":389,"threads_started":1}
I20260806 09:10:01.113431 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.085108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:33959 (local address 127.0.0.1:52608)
0806 09:10:01.086108 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.086172 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.089813 (+ 3641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.090933 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.090955 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.091977 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.093153 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.093193 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.094772 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.094782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.097421 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.097450 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.098645 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.098657 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.099107 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.100125 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.100171 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.103965 (+ 3794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.108130 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.108151 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.108170 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.109082 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.109625 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.109630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.109634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.109915 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.110612 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.110622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.111093 (+ 471us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.111411 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.112045 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":379,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.188@KRBTEST.COM" created.
I20260806 09:10:01.115870 18944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:01.136897 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.116820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43300)
0806 09:10:01.117045 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.117075 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.117205 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.118609 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.118613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.119139 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.119304 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.119313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.121637 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.121642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.122615 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.122630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.124280 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.124287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.124463 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.125279 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.125299 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.128223 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.133004 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.133012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.133019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.133693 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.134283 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.134290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.134297 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.134497 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.135376 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.135436 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.135692 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.135889 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.136280 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 09:10:01.136916 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.117179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.27.187.189:45115)
0806 09:10:01.117932 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.117945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.117985 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:10:01.118074 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.118085 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.118390 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.118589 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.119534 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.121430 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.122838 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.123970 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.124155 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.128513 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.128535 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.128560 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.128614 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.132796 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.133917 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.133927 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.133930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.134126 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.134727 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.134734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.134738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.135219 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.135573 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.136050 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.136464 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260806 09:10:01.140400 18944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:01.141914 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.141338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46152)
0806 09:10:01.141547 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.141736 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 09:10:01.142344 18892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.144236 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.143807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46162)
0806 09:10:01.143999 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.144108 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 09:10:01.144471 18944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
I20260806 09:10:01.164136 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:39079
--webserver_interface=127.27.187.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:10:01.185218 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } attempt: 5
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:01.193786 18944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 1
I20260806 09:10:01.195247 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.194556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46166)
0806 09:10:01.194869 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.195004 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 09:10:01.197063 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.196505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46170)
0806 09:10:01.196746 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.196875 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 09:10:01.197307 18944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.202416 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.186117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43302)
0806 09:10:01.186345 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.186373 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.186533 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.186942 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.186947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.187417 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.187617 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.187628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.188709 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.188715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.189611 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.189624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.190432 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.190438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.190666 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.191429 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.191450 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.194472 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.199464 (+ 4992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.199469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.199473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.199911 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.200573 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.200579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.200582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.200759 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.201419 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.201423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.201575 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.201727 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.202053 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:10:01.202531 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.186332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.27.187.189:45115)
0806 09:10:01.186547 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.186554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.186578 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:10:01.186712 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.186716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.186781 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.186898 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.187795 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.188553 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.189785 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.190263 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.190409 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.194785 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.194793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.194798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.194841 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.199255 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.200061 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.200069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.200076 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.200302 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.200881 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.200887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.200890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.201289 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.201442 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.201854 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.202094 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:10:01.203903 18869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:01.205268 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.204678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46178)
0806 09:10:01.204937 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.205085 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:10:01.206542 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.206139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46194)
0806 09:10:01.206301 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.206391 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:10:01.206794 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.258078 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 1
I20260806 09:10:01.259797 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.259055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46210)
0806 09:10:01.259407 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.259544 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260806 09:10:01.261317 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.260794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46214)
0806 09:10:01.261005 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.261121 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 09:10:01.261610 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.265585 18944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 2
I20260806 09:10:01.267122 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.266428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46226)
0806 09:10:01.266808 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.266916 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 09:10:01.268508 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.268007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46228)
0806 09:10:01.268283 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.268364 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260806 09:10:01.268739 18944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.329895 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 2
I20260806 09:10:01.331326 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.330736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46230)
0806 09:10:01.330982 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.331099 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:10:01.332589 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.332083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46236)
0806 09:10:01.332331 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.332440 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260806 09:10:01.332894 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.359977 18944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 3
I20260806 09:10:01.361393 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.360801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46244)
0806 09:10:01.361029 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.361203 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 09:10:01.362754 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.362207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46256)
0806 09:10:01.362466 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.362551 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260806 09:10:01.363121 18944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.439302 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 3
I20260806 09:10:01.440769 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.440083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46266)
0806 09:10:01.440384 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.440520 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:10:01.442222 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.441643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46280)
0806 09:10:01.441940 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.442056 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260806 09:10:01.442536 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.506420 18944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 4
I20260806 09:10:01.507884 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.507282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46282)
0806 09:10:01.507577 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.507688 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:10:01.509327 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.508861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46292)
0806 09:10:01.509053 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.509160 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 09:10:01.509609 18944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
W20260806 09:10:01.542833 18952 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:01.543217 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:01.543292 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:01.543380 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:01.551276 18952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:01.551394 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:01.551453 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:01.551499 18952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:01.551564 18952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:01.562824 18952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:33959,127.27.187.189:45115,127.27.187.188:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.187.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:01.564325 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:01.565706 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:01.579473 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:01.579577 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:01.579625 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:01.580753 18952 server_base.cc:1034] running on GCE node
I20260806 09:10:01.581633 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:01.583287 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:01.584553 18952 hybrid_clock.cc:648] HybridClock initialized: now 1786007401584489 us; error 43 us; skew 500 ppm
I20260806 09:10:01.584926 18952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:01.588860 18952 init.cc:377] Logged in from keytab as kudu/127.27.187.188@KRBTEST.COM (short username kudu)
I20260806 09:10:01.590654 18952 webserver.cc:460] Webserver started at http://127.27.187.188:37345/ using document root <none> and password file <none>
I20260806 09:10:01.591517 18952 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:01.591673 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:01.592053 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:01.595590 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data/instance:
uuid: "72da4d9b232b4bfbadc1b2698fabd889"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "c958500b3c6981e22c2848514a20a9bd"
server_key_iv: "d2578633eb016b798c3443a852fd59cc"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.596527 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal/instance:
uuid: "72da4d9b232b4bfbadc1b2698fabd889"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "c958500b3c6981e22c2848514a20a9bd"
server_key_iv: "d2578633eb016b798c3443a852fd59cc"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.602596 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:10:01.607153 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:01.608914 18952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:10:01.609156 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "72da4d9b232b4bfbadc1b2698fabd889"
format_stamp: "Formatted at 2026-08-06 09:10:01 on dist-test-slave-8tdl"
server_key: "c958500b3c6981e22c2848514a20a9bd"
server_key_iv: "d2578633eb016b798c3443a852fd59cc"
server_key_version: "encryptionkey@0"
I20260806 09:10:01.609424 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:01.619791 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 4
I20260806 09:10:01.620976 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.620492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46300)
0806 09:10:01.620710 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.620820 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 09:10:01.622330 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.621805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46310)
0806 09:10:01.622005 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.622170 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 09:10:01.622651 18869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:39079: Network error: Client connection negotiation failed: client connection to 127.27.187.188:39079: connect: Connection refused (error 111)
I20260806 09:10:01.633665 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:01.639302 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:01.639622 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:01.661650 18952 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.188:39079
I20260806 09:10:01.661751 19019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.188:39079 every 8 connection(s)
I20260806 09:10:01.663261 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
I20260806 09:10:01.667673 19020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:01.672580 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 18952
I20260806 09:10:01.672798 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007003447047-28398-0/minicluster-data/master-2/wal/instance
I20260806 09:10:01.673324 28398 external_mini_cluster.cc:1428] Setting key e3727a211643abc80602627b600a8397
I20260806 09:10:01.675737 19020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:01.696931 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.679480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.27.187.190:33959)
0806 09:10:01.680872 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.680877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.680894 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:10:01.680945 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.680950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.681008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.681139 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.682912 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.683624 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.685192 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.685617 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.685762 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.690303 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.690309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.690313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.690360 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.693486 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.694292 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.694310 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.694314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.694457 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.695025 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.695030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.695036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.695334 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.695499 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.696233 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.696482 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":1104,"threads_started":1}
I20260806 09:10:01.697992 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.675730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.27.187.188:39079)
0806 09:10:01.676768 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.676787 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.676845 (+ 58us) server_negotiation.cc:407] Connection header received
0806 09:10:01.679248 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.679273 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.679974 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.682096 (+ 2122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.682364 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.683729 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.685274 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.685911 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.686108 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.690983 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.690992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.691007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.691043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.693873 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.694636 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.694643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.694645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.694868 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.695266 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.695271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.695274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.695858 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.696196 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.696550 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.696851 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":765,"thread_start_us":507,"threads_started":1}
I20260806 09:10:01.698037 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.679381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:33959 (local address 127.0.0.1:52612)
0806 09:10:01.680226 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.680280 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.680435 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.681231 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.681243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.681914 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.682644 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.682660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.683815 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.683819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.685007 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.685019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.685889 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.685897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.686104 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.686978 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.687005 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.690043 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.693660 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.693674 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.693687 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.694155 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.694576 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.694581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.694585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.694908 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.695477 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.695485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.695846 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.696145 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.696722 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":49,"spinlock_wait_cycles":18304,"thread_start_us":334,"threads_started":1}
I20260806 09:10:01.699927 19020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:01.717943 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.700626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:45115 (local address 127.0.0.1:43314)
0806 09:10:01.700820 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.700841 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.700970 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.701316 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.701320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.701758 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.701965 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.701977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.703307 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.703313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.704533 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.704550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.705502 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.705506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.705655 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.706329 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.706348 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.709332 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.714936 (+ 5604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.714941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.714944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.715437 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.715930 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.715938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.715946 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.716120 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.716774 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.716780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.716959 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.717153 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.717510 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 09:10:01.717978 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.700780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.27.187.189:45115)
0806 09:10:01.700959 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.700964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.700983 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:10:01.701085 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.701089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.701153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.701287 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.702207 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.703159 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.704722 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.705311 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.705467 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.709649 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.709661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.709668 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.709719 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.714735 (+ 5016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.715605 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.715612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.715616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.715788 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.716255 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.716261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.716265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.716632 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.716813 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.717270 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.717546 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:10:01.719135 19020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:01.723776 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.702130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.27.187.190:33959)
0806 09:10:01.702350 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.702356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.702373 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:10:01.702435 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.702439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.702522 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.702628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.703983 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.704943 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.706758 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.707476 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.707642 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.713622 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.713633 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.713638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.713706 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.717036 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.717864 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.717871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.717874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.718047 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.719187 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.719191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.719194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.719711 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.720017 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.723042 (+ 3025us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.723389 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:01.738453 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.720235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.27.187.188:39079)
0806 09:10:01.720446 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.720453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.720832 (+ 379us) server_negotiation.cc:407] Connection header received
0806 09:10:01.721073 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.721080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.721304 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.721396 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.723471 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.724379 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.726313 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.727143 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.727341 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.731684 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.731691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.731696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.731740 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.735489 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.736235 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.736243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.736248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.736420 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.736861 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.736868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.736875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.737224 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.737394 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.737760 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.738131 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":59}
I20260806 09:10:01.738461 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.720499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46338)
0806 09:10:01.720712 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.720736 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.720940 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.721963 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.721972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.722999 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.723251 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.723266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.724524 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.724532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.726088 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.726123 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.727313 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.727319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.727556 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.728350 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.728376 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.731431 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.735682 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.735686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.735689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.736097 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.736577 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.736584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.736587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.736734 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.737368 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.737373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.737514 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.737670 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.737999 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 09:10:01.743819 19020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: Bootstrap starting.
I20260806 09:10:01.745167 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.726586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.27.187.189:45115)
0806 09:10:01.726836 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.726842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.726971 (+ 129us) server_negotiation.cc:407] Connection header received
0806 09:10:01.727238 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.727246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.727349 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.727450 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.729500 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.730809 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.732614 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.733265 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.733378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.739088 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.739094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.739097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.739138 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.742021 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.742917 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.742925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.742930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.743131 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.743584 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.743589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.743591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.743880 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.744148 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.744648 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.744844 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:10:01.747905 19020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:01.749281 19020 log.cc:824] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:01.751272 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:10:01.752751 19020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: No bootstrap required, opened a new log
I20260806 09:10:01.754765 18844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "35f41a63d4d144ab998877d6b23c64a0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52622
I20260806 09:10:01.764292 19020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.764724 19020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da4d9b232b4bfbadc1b2698fabd889, State: Initialized, Role: FOLLOWER
I20260806 09:10:01.765336 19020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.766486 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:01.766965 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:01.767927 19020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:10:01.780329 19039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:10:01.780506 19039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:10:01.777429 18952 master_runner.cc:186] Error getting master registration for 127.27.187.190:33959: OK, instance_id { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" instance_seqno: 1786007400465243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:10:01.786516 18952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:10:01.808887 18944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 5
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:01.825850 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.810064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.27.187.188:39079)
0806 09:10:01.810357 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.810362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.810378 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:10:01.810431 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.810435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.810500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.810774 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.811627 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.812462 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.813897 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.814747 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.814878 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.818380 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.818386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.818389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.818428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.822636 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.823487 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.823493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.823497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.823664 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.824079 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.824084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.824087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.824570 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.824736 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.825218 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.825448 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:10:01.825871 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.809767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46340)
0806 09:10:01.809994 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.810016 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.810172 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.810800 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.810806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.811277 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.811448 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.811459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.812641 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.812648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.813744 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.813759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.814910 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.814916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.815172 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.815883 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.815903 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.818138 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.822886 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.822892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.822896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.823345 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.823809 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.823815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.823821 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.823962 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.824715 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.824721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.824923 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.825084 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.825429 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:10:01.831446 18944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: Bootstrap starting.
I20260806 09:10:01.835530 18944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:01.836758 18944 log.cc:824] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:01.840122 18944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: No bootstrap required, opened a new log
I20260806 09:10:01.848881 18944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.849252 18944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b128eceebbc4528a0e4ddf4f7a84f1e, State: Initialized, Role: FOLLOWER
I20260806 09:10:01.849779 18944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.850940 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:01.851387 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:01.852289 18944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:10:01.863844 19054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:10:01.864261 19054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:10:01.860715 18876 master_runner.cc:186] Error getting master registration for 127.27.187.190:33959: OK, instance_id { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" instance_seqno: 1786007400465243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:10:01.870766 18876 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:10:01.892064 18974 tablet.cc:2369] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:01.899941 18869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } attempt: 5
Aug 06 09:10:01 dist-test-slave-8tdl krb5kdc[18789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:01.920819 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.900794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:39079 (local address 127.0.0.1:46348)
0806 09:10:01.901104 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:01.901126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:01.901273 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:01.901763 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:01.901769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:01.902348 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:01.902604 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.902619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.904228 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.904235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.905766 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:01.905788 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.907112 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.907119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:01.907350 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.908251 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:01.908282 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:01.911812 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:01.917371 (+ 5559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.917379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.917384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.918040 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.918591 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:01.918597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:01.918601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:01.918848 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:01.919551 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:01.919557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:01.919805 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 09:10:01.919981 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 09:10:01.920363 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260806 09:10:01.920876 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:01.900877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.27.187.188:39079)
0806 09:10:01.901101 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:01.901107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:01.901396 (+ 289us) server_negotiation.cc:407] Connection header received
0806 09:10:01.901473 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:01.901479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:01.901592 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:01.901696 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:01.902870 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.904052 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.905990 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:01.906903 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:01.907065 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:01.912139 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:01.912149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:01.912154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:01.912217 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:01.917154 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.918231 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.918238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.918243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.918459 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:01.918982 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:01.918988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:01.918992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:01.919413 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:01.919602 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:01.920132 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 09:10:01.920421 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:10:01.928028 18869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0: Bootstrap starting.
I20260806 09:10:01.933346 18869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:01.935048 18869 log.cc:824] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:01.942456 18869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0: No bootstrap required, opened a new log
I20260806 09:10:01.958139 18869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.959213 18869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f41a63d4d144ab998877d6b23c64a0, State: Initialized, Role: FOLLOWER
I20260806 09:10:01.960371 18869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.962196 19057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:01.962885 19057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:01.965384 18844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:10:01.965639 18844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:01.965891 18869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:01.969751 18844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.972623 18918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35f41a63d4d144ab998877d6b23c64a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e"
I20260806 09:10:01.973213 18918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:01.977756 18918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f41a63d4d144ab998877d6b23c64a0 in term 1.
I20260806 09:10:01.978698 18821 leader_election.cc:302] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f41a63d4d144ab998877d6b23c64a0, 7b128eceebbc4528a0e4ddf4f7a84f1e; no voters:
I20260806 09:10:01.979331 19057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:01.981045 18995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35f41a63d4d144ab998877d6b23c64a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "72da4d9b232b4bfbadc1b2698fabd889"
I20260806 09:10:01.981684 18995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:01.985572 18995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f41a63d4d144ab998877d6b23c64a0 in term 1.
I20260806 09:10:01.986574 18844 leader_election.cc:288] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [CANDIDATE]: Term 1 election: Requested vote from peers 7b128eceebbc4528a0e4ddf4f7a84f1e (127.27.187.189:45115), 72da4d9b232b4bfbadc1b2698fabd889 (127.27.187.188:39079)
I20260806 09:10:01.987020 19057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35f41a63d4d144ab998877d6b23c64a0, State: Running, Role: LEADER
I20260806 09:10:01.987804 19057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } }
I20260806 09:10:01.996691 19059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f41a63d4d144ab998877d6b23c64a0. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:01.997404 19059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:02.011722 19062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:02.035755 19062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 09:10:02.049829 18897 tablet.cc:2369] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:02.078783 18995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [term 1 FOLLOWER]: Refusing update from remote peer 35f41a63d4d144ab998877d6b23c64a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:02.080397 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:02.015343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.27.187.188:39079)
0806 09:10:02.015566 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.015573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.024890 (+ 9317us) server_negotiation.cc:407] Connection header received
0806 09:10:02.025443 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.025450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.025554 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.025637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.028940 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.030327 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.037832 (+ 7505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.038562 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.038720 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:02.053257 (+ 14537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:02.053266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:02.053271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:02.053322 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:02.057706 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.065705 (+ 7999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.065713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.065717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.065953 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.066576 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.066585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.066589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.067054 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:02.070952 (+ 3898us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:02.079643 (+ 8691us) server_negotiation.cc:299] Negotiation successful
0806 09:10:02.079973 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:10:02.080417 19059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:02.098042 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:02.023275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.27.187.190:33959)
0806 09:10:02.023497 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.023505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.028646 (+ 5141us) server_negotiation.cc:407] Connection header received
0806 09:10:02.030830 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.030838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.030947 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.031034 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.040533 (+ 9499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.041776 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.050906 (+ 9130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.051709 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.051832 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:02.085738 (+ 33906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:02.085747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:02.085753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:02.085807 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:02.090400 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.094984 (+ 4584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.094992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.094996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.095248 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.095802 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.095809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.095812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.096285 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:02.096606 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:02.097264 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 09:10:02.097581 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:10:02.101893 18918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [term 1 FOLLOWER]: Refusing update from remote peer 35f41a63d4d144ab998877d6b23c64a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:02.103281 19059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:02.115746 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f41a63d4d144ab998877d6b23c64a0. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.116168 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:02.118983 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:02.034652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.27.187.189:45115)
0806 09:10:02.034887 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.034894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.047519 (+ 12625us) server_negotiation.cc:407] Connection header received
0806 09:10:02.048011 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.048017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.048121 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.048204 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.061458 (+ 13254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.062791 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.083485 (+ 20694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:02.084313 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:02.084434 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:02.107032 (+ 22598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:02.107041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:02.107047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:02.107117 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:02.111768 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.112919 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.112926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.112930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.113303 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:02.114406 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:02.114413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:02.114416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:02.114887 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:02.115176 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:02.118232 (+ 3056us) server_negotiation.cc:299] Negotiation successful
0806 09:10:02.118558 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:10:02.128722 19072 mvcc.cc:205] Tried to move back new op lower bound from 7315486318864408576 to 7315486318548770816. Current Snapshot: MvccSnapshot[applied={T|T < 7315486318864408576}]
I20260806 09:10:02.139951 19074 mvcc.cc:205] Tried to move back new op lower bound from 7315486318864408576 to 7315486318548770816. Current Snapshot: MvccSnapshot[applied={T|T < 7315486318864408576}]
I20260806 09:10:02.141891 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f41a63d4d144ab998877d6b23c64a0. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.142352 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:02.143110 19059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.143411 19059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:02.145718 19062 catalog_manager.cc:1347] Generated new cluster ID: a654b65b011e4627958959663b7a2511
I20260806 09:10:02.145835 19062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:02.160988 19057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.161494 19057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:02.162047 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.162335 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:02.148731 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35f41a63d4d144ab998877d6b23c64a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f41a63d4d144ab998877d6b23c64a0" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 33959 } } peers { permanent_uuid: "7b128eceebbc4528a0e4ddf4f7a84f1e" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 45115 } } peers { permanent_uuid: "72da4d9b232b4bfbadc1b2698fabd889" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 39079 } } }
I20260806 09:10:02.162904 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:02.171737 19087 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:10:02.176347 19079 mvcc.cc:205] Tried to move back new op lower bound from 7315486318864408576 to 7315486318548770816. Current Snapshot: MvccSnapshot[applied={T|T < 7315486318864408576 or (T in {7315486318864408576})}]
I20260806 09:10:02.196393 19062 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:02.201650 19062 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:02.228979 19062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35f41a63d4d144ab998877d6b23c64a0: Generated new TSK 0
I20260806 09:10:02.230422 19062 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:10:02.350585 28398 mini_kdc.cc:348] Time spent kdestroy: real 0.105s user 0.000s sys 0.034s
W20260806 09:10:02.398757 19023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.385453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.27.187.188:39079)
0806 09:10:02.385680 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.385690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.396904 (+ 11214us) server_negotiation.cc:407] Connection header received
0806 09:10:02.397142 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.397148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.397247 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.397378 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.398436 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46378: BlockingRecv error: recv got EOF from 127.0.0.1:46378 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 09:10:02.401438 19025 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.393373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.27.187.190:33959)
0806 09:10:02.393675 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.393681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.399658 (+ 5977us) server_negotiation.cc:407] Connection header received
0806 09:10:02.399870 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.399876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.399977 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.400093 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.401157 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52644: BlockingRecv error: recv got EOF from 127.0.0.1:52644 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 09:10:02.405135 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.402626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.27.187.189:45115)
0806 09:10:02.402898 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.402906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.403387 (+ 481us) server_negotiation.cc:407] Connection header received
0806 09:10:02.403660 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.403666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.403759 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.403834 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.404859 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 09:10:02.446210 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.436394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.27.187.189:45115)
0806 09:10:02.436613 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.436620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.443707 (+ 7087us) server_negotiation.cc:407] Connection header received
0806 09:10:02.444337 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.444349 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.444516 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.444603 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.445861 (+ 1258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 09:10:02.488958 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.478608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.27.187.189:45115)
0806 09:10:02.478873 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.478880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.486534 (+ 7654us) server_negotiation.cc:407] Connection header received
0806 09:10:02.487173 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.487180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.487292 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.487379 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.488662 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43358: BlockingRecv error: recv got EOF from 127.0.0.1:43358 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260806 09:10:02.522382 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.512743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.27.187.189:45115)
0806 09:10:02.512937 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.512943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.520294 (+ 7351us) server_negotiation.cc:407] Connection header received
0806 09:10:02.520901 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.520908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.521037 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.521120 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.522112 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43372: BlockingRecv error: recv got EOF from 127.0.0.1:43372 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 09:10:02.535557 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.532988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.27.187.189:45115)
0806 09:10:02.533242 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.533250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.533936 (+ 686us) server_negotiation.cc:407] Connection header received
0806 09:10:02.534174 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.534180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.534280 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.534408 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.535338 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43380: BlockingRecv error: recv got EOF from 127.0.0.1:43380 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260806 09:10:02.550567 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.548084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.27.187.189:45115)
0806 09:10:02.548284 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.548291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.548980 (+ 689us) server_negotiation.cc:407] Connection header received
0806 09:10:02.549254 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.549260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.549353 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.549468 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.550282 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43382: BlockingRecv error: recv got EOF from 127.0.0.1:43382 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 09:10:02.572989 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.570378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.27.187.189:45115)
0806 09:10:02.570699 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.570708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.571341 (+ 633us) server_negotiation.cc:407] Connection header received
0806 09:10:02.571607 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.571614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.571741 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.571864 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.572735 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43386: BlockingRecv error: recv got EOF from 127.0.0.1:43386 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260806 09:10:02.611006 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.608588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.27.187.189:45115)
0806 09:10:02.608860 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.608870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.609428 (+ 558us) server_negotiation.cc:407] Connection header received
0806 09:10:02.609677 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.609689 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.609777 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.609893 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.610724 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43394: BlockingRecv error: recv got EOF from 127.0.0.1:43394 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260806 09:10:02.681890 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.679142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.27.187.189:45115)
0806 09:10:02.679463 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.679473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.680221 (+ 748us) server_negotiation.cc:407] Connection header received
0806 09:10:02.680489 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.680494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.680586 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.680712 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.681646 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43400: BlockingRecv error: recv got EOF from 127.0.0.1:43400 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 09:10:02.782974 19039 catalog_manager.cc:1259] Loaded cluster ID: a654b65b011e4627958959663b7a2511
I20260806 09:10:02.783087 19039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: loading cluster ID for follower catalog manager: success
I20260806 09:10:02.785825 19039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: acquiring CA information for follower catalog manager: success
I20260806 09:10:02.787946 19039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72da4d9b232b4bfbadc1b2698fabd889: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:10:02.818079 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:02.815981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.27.187.189:45115)
0806 09:10:02.816261 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:02.816268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:02.816524 (+ 256us) server_negotiation.cc:407] Connection header received
0806 09:10:02.816667 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:02.816671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:02.816755 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:02.816860 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:02.817859 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43410: BlockingRecv error: recv got EOF from 127.0.0.1:43410 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 09:10:02.867286 19054 catalog_manager.cc:1259] Loaded cluster ID: a654b65b011e4627958959663b7a2511
I20260806 09:10:02.867496 19054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: loading cluster ID for follower catalog manager: success
I20260806 09:10:02.870496 19054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: acquiring CA information for follower catalog manager: success
I20260806 09:10:02.872782 19054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b128eceebbc4528a0e4ddf4f7a84f1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:10:03.086705 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:03.078763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.27.187.189:45115)
0806 09:10:03.079045 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:03.079053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:03.079501 (+ 448us) server_negotiation.cc:407] Connection header received
0806 09:10:03.079785 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:03.079790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:03.079958 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:03.080065 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:03.080681 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.081504 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.083945 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.084461 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.084575 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:03.085320 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:03.085924 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:03.086048 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:03.086219 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 09:10:03.086324 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:10:03.101922 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:03.094410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.27.187.188:39079)
0806 09:10:03.094976 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:03.094981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:03.095004 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:10:03.095164 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:03.095168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:03.095346 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:03.095523 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:03.096037 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.096838 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.098533 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.099345 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.099488 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:03.100428 (+ 940us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:03.101183 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:03.101315 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:03.101537 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 09:10:03.101624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":317,"threads_started":1}
I20260806 09:10:03.121671 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:03.110459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.27.187.188:39079)
0806 09:10:03.110662 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:03.110668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:03.113772 (+ 3104us) server_negotiation.cc:407] Connection header received
0806 09:10:03.114154 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:03.114159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:03.114296 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:03.114426 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:03.115364 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.116558 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.119126 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.119808 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.119933 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:03.120612 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:03.120764 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:03.120871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:03.121129 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 09:10:03.121243 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:10:03.121910 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:03.111219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.27.187.189:45115)
0806 09:10:03.111456 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:03.111465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:03.114125 (+ 2660us) server_negotiation.cc:407] Connection header received
0806 09:10:03.114552 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:03.114567 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:03.114706 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:03.114812 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:03.115643 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.116672 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.118948 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.119653 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.119820 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:03.120659 (+ 839us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:03.120788 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:03.120875 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:03.121471 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 09:10:03.121565 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:10:03.123041 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:03.110015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.27.187.190:33959)
0806 09:10:03.114996 (+ 4981us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:03.115004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:03.115029 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:10:03.115147 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:03.115151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:03.115368 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:03.116243 (+ 875us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:03.116418 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.117517 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.119996 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:03.120782 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:03.121044 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:03.121613 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:03.122209 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:03.122444 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:03.122622 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 09:10:03.122751 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4789,"thread_start_us":4674,"threads_started":1}
I20260806 09:10:03.130828 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18802
I20260806 09:10:03.132234 18864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:03.247821 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18802
I20260806 09:10:03.274507 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18876
I20260806 09:10:03.275839 18938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:03.388700 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18876
I20260806 09:10:03.412853 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 18952
I20260806 09:10:03.414502 19015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:03.526821 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 18952
2026-08-06T09:10:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5137 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:03 dist-test-slave-8tdl krb5kdc[19169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:03 dist-test-slave-8tdl krb5kdc[19169](info): set up 2 sockets
Aug 06 09:10:03 dist-test-slave-8tdl krb5kdc[19169](info): commencing operation
krb5kdc: starting...
W20260806 09:10:04.730206 28398 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/test-user.keytab.
Aug 06 09:10:04 dist-test-slave-8tdl krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.187.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
Entry for principal kudu/127.27.187.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab.
I20260806 09:10:05.112368 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:42381
--webserver_interface=127.27.187.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:05.485430 19182 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:05.485829 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:05.486012 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:05.486150 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:05.494196 19182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:05.494323 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:05.494426 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:05.494522 19182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:05.494638 19182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:05.505604 19182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.190.keytab
--principal=kudu/127.27.187.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.190:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.187.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:05.507308 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:05.509126 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:05.523516 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:05.523506 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:05.523542 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:05.524981 19182 server_base.cc:1034] running on GCE node
I20260806 09:10:05.525960 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:05.527652 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:05.528942 19182 hybrid_clock.cc:648] HybridClock initialized: now 1786007405528848 us; error 64 us; skew 500 ppm
I20260806 09:10:05.529407 19182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:05 dist-test-slave-8tdl krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:05.533288 19182 init.cc:377] Logged in from keytab as kudu/127.27.187.190@KRBTEST.COM (short username kudu)
I20260806 09:10:05.535225 19182 webserver.cc:460] Webserver started at http://127.27.187.190:36819/ using document root <none> and password file <none>
I20260806 09:10:05.536023 19182 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:05.536175 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:05.536514 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:05.540411 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data/instance:
uuid: "e3d75c4e03fa407799ddd8c66672002e"
format_stamp: "Formatted at 2026-08-06 09:10:05 on dist-test-slave-8tdl"
server_key: "9ff84780ed539d1b0ae49a3429fbb89a"
server_key_iv: "d6c41101a02754ffb7fb1aa13fd4fb0d"
server_key_version: "encryptionkey@0"
I20260806 09:10:05.541392 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance:
uuid: "e3d75c4e03fa407799ddd8c66672002e"
format_stamp: "Formatted at 2026-08-06 09:10:05 on dist-test-slave-8tdl"
server_key: "9ff84780ed539d1b0ae49a3429fbb89a"
server_key_iv: "d6c41101a02754ffb7fb1aa13fd4fb0d"
server_key_version: "encryptionkey@0"
I20260806 09:10:05.547825 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:10:05.552470 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:05.554466 19182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:10:05.554786 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal
uuid: "e3d75c4e03fa407799ddd8c66672002e"
format_stamp: "Formatted at 2026-08-06 09:10:05 on dist-test-slave-8tdl"
server_key: "9ff84780ed539d1b0ae49a3429fbb89a"
server_key_iv: "d6c41101a02754ffb7fb1aa13fd4fb0d"
server_key_version: "encryptionkey@0"
I20260806 09:10:05.555068 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:05.593710 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:05.599354 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:05.599733 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:05.621637 19182 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.190:42381
I20260806 09:10:05.621680 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.190:42381 every 8 connection(s)
I20260806 09:10:05.623234 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:05.627709 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:05.630769 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 19182
I20260806 09:10:05.631004 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-0/wal/instance
I20260806 09:10:05.631500 28398 external_mini_cluster.cc:1428] Setting key b5d26daac779b73120ceb01e03d192b0
I20260806 09:10:05.635397 19250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:05 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:05.660868 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.639081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.27.187.190:42381)
0806 09:10:05.640013 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:05.640033 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:05.640522 (+ 489us) server_negotiation.cc:407] Connection header received
0806 09:10:05.642748 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:05.642754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:05.643068 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:05.643299 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:05.644739 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:05.646002 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:05.647604 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:05.648204 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:05.648417 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:05.652100 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:05.652111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:05.652126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:05.652157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:05.655839 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:05.656627 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:05.656633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:05.656636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:05.656802 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:05.657232 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:05.657239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:05.657247 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:05.657806 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:05.658034 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:05.658594 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 09:10:05.659353 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":258,"server-negotiator.queue_time_us":671,"thread_start_us":418,"threads_started":1}
I20260806 09:10:05.660913 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.639092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42381 (local address 127.0.0.1:43610)
0806 09:10:05.640349 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.640385 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:05.642265 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:05.643313 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:05.643319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:05.644093 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:05.644473 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:05.644494 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:05.646197 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:05.646203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:05.647450 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:05.647463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:05.648457 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:05.648464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:05.648637 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:05.649375 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:05.649398 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:05.651825 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:05.656007 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:05.656020 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:05.656034 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:05.656459 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:05.656921 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:05.656927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:05.656930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:05.657112 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:05.657953 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:05.657963 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:05.658193 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 09:10:05.658468 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 09:10:05.659240 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":144,"spinlock_wait_cycles":17152,"thread_start_us":789,"threads_started":1}
WARNING: no policy specified for kudu/127.27.187.189@KRBTEST.COM; defaulting to no policy
I20260806 09:10:05.664657 19250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:05.666217 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.665572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36072)
0806 09:10:05.665912 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.666065 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260806 09:10:05.666585 19201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.189@KRBTEST.COM" created.
I20260806 09:10:05.668780 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.668299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36080)
0806 09:10:05.668489 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.668589 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:10:05.669155 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:41331: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
Entry for principal kudu/127.27.187.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab.
I20260806 09:10:05.714926 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:41331
--webserver_interface=127.27.187.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:10:05.718636 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } attempt: 1
I20260806 09:10:05.720110 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.719536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36096)
0806 09:10:05.719765 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.719927 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:10:05.721526 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.720991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36112)
0806 09:10:05.721212 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.721325 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 09:10:05.721894 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:41331: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
I20260806 09:10:05.790246 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } attempt: 2
I20260806 09:10:05.791764 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.791207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36114)
0806 09:10:05.791499 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.791616 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 09:10:05.792992 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.792577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36128)
0806 09:10:05.792766 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.792842 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:10:05.793210 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:41331: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
I20260806 09:10:05.884518 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } attempt: 3
I20260806 09:10:05.885826 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.885335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36134)
0806 09:10:05.885570 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.885681 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:10:05.887082 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:05.886588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36142)
0806 09:10:05.886813 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:05.886900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:10:05.887398 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:41331: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
I20260806 09:10:06.030791 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } attempt: 4
I20260806 09:10:06.032188 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.031570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36152)
0806 09:10:06.031915 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.032016 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260806 09:10:06.033741 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.033146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36162)
0806 09:10:06.033394 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.033511 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 09:10:06.034093 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.189:41331: Network error: Client connection negotiation failed: client connection to 127.27.187.189:41331: connect: Connection refused (error 111)
W20260806 09:10:06.151324 19257 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:06.151731 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:06.151847 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:06.151995 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:06.161088 19257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:06.161216 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:06.161280 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:06.161408 19257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:06.161556 19257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:06.173666 19257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.189.keytab
--principal=kudu/127.27.187.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.189:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.187.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:06.175377 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:06.176831 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:06.190747 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:06.190814 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:06.190779 19263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:06.191287 19257 server_base.cc:1034] running on GCE node
I20260806 09:10:06.192397 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:06.194104 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:06.195428 19257 hybrid_clock.cc:648] HybridClock initialized: now 1786007406195325 us; error 66 us; skew 500 ppm
I20260806 09:10:06.195868 19257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:06.199810 19257 init.cc:377] Logged in from keytab as kudu/127.27.187.189@KRBTEST.COM (short username kudu)
I20260806 09:10:06.201718 19257 webserver.cc:460] Webserver started at http://127.27.187.189:36603/ using document root <none> and password file <none>
I20260806 09:10:06.202538 19257 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:06.202754 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:06.203132 19257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:06.206593 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data/instance:
uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "738f1e2c2dccf7394d0e827216d4bdd2"
server_key_iv: "8e59a5a76d769ac393a1f5c0e0ba2785"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.207628 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal/instance:
uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "738f1e2c2dccf7394d0e827216d4bdd2"
server_key_iv: "8e59a5a76d769ac393a1f5c0e0ba2785"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.213593 19257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:10:06.218037 19273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:06.219827 19257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:10:06.220080 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal
uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "738f1e2c2dccf7394d0e827216d4bdd2"
server_key_iv: "8e59a5a76d769ac393a1f5c0e0ba2785"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.220357 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:06.248867 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:06.254225 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:06.254562 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:06.276873 19257 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.189:41331
I20260806 09:10:06.276842 19324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.189:41331 every 8 connection(s)
I20260806 09:10:06.278477 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/data/info.pb
I20260806 09:10:06.283396 19325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:06.287297 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 19257
I20260806 09:10:06.287536 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-1/wal/instance
I20260806 09:10:06.288018 28398 external_mini_cluster.cc:1428] Setting key 59a5340607e6dd136724a8583cfe97f8
I20260806 09:10:06.291425 19325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:06.316959 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.294580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.27.187.190:42381)
0806 09:10:06.295354 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.295360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.296953 (+ 1593us) server_negotiation.cc:407] Connection header received
0806 09:10:06.299324 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.299330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.299437 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.299565 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.301365 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.302154 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.304317 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.304874 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.305040 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.309454 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.309460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.309464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.309517 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.312925 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.314018 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.314024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.314026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.314260 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.314859 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.314863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.314866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.315212 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.315417 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.316245 (+ 828us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.316509 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":418,"threads_started":1}
I20260806 09:10:06.318037 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.295630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42381 (local address 127.0.0.1:43616)
0806 09:10:06.296742 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.296790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.298978 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.299859 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.299869 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.300472 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.301105 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.301125 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.302353 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.302358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.304138 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.304154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.305142 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.305150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.305431 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.306297 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.306323 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.309113 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.313219 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.313229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.313246 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.313843 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.314410 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.314415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.314419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.314660 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.315365 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.315374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.315827 (+ 453us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.316121 (+ 294us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.316750 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":430,"threads_started":1}
I20260806 09:10:06.320320 19325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.187.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.187.188@KRBTEST.COM" created.
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:06.333411 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } attempt: 5
I20260806 09:10:06.341080 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.321362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36174)
0806 09:10:06.321612 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.321632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.321777 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.323028 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.323035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.323440 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.323702 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.323724 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.325507 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.325512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.326905 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.326919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.328494 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.328498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.328705 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.329454 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.329474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.332405 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.337369 (+ 4964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.337378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.337389 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.338095 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.338567 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.338575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.338581 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.338817 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.339647 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.339655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.339867 (+ 212us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.340055 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.340507 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 09:10:06.341106 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.321630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.27.187.189:41331)
0806 09:10:06.322410 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.322417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.322446 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:10:06.322496 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.322502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.322842 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.322977 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.323914 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.325356 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.327132 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.328303 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.328482 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.332697 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.332721 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.332748 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.332805 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.337179 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.338255 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.338265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.338269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.338430 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.338945 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.338950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.338953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.339488 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.339736 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.340275 (+ 539us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.340643 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":614,"thread_start_us":453,"threads_started":1}
I20260806 09:10:06.344614 19325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:06.346127 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.345444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58052)
0806 09:10:06.345705 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.345864 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 09:10:06.346467 19277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.348683 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.348101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58056)
0806 09:10:06.348331 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.348477 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 09:10:06.349037 19325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.353120 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.334255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36176)
0806 09:10:06.334606 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.334631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.334789 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.337852 (+ 3063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.337859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.338401 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.338659 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.338670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.340364 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.340370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.341776 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.341790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.342587 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.342594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.342845 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.343560 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.343580 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.345984 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.350266 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.350273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.350277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.350827 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.351241 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.351247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.351250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.351408 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.351977 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.351982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.352194 (+ 212us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.352350 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.352737 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260806 09:10:06.353164 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.334469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.27.187.189:41331)
0806 09:10:06.337513 (+ 3044us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.337517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.337533 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:10:06.337586 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.337589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.337653 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.337782 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.338874 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.340181 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.341921 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.342411 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.342555 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.346487 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.346499 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.346505 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.346563 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.350085 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.350972 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.350977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.350980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.351115 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.351543 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.351557 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.351559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.351848 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.352038 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.352461 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.352711 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2920,"thread_start_us":1242,"threads_started":1}
I20260806 09:10:06.354250 19250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:06.355505 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.354921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58072)
0806 09:10:06.355158 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.355285 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 09:10:06.356940 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.356351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58080)
0806 09:10:06.356641 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.356759 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 09:10:06.357302 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
Entry for principal kudu/127.27.187.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab.
I20260806 09:10:06.379036 28398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
/tmp/dist-test-taskvYHTS_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:37163
--webserver_interface=127.27.187.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.187.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:10:06.398481 19325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 1
I20260806 09:10:06.399969 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.399337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58086)
0806 09:10:06.399596 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.399747 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:10:06.401206 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.400746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58090)
0806 09:10:06.400937 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.401045 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 09:10:06.401443 19325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.408602 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 1
I20260806 09:10:06.410115 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.409424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58096)
0806 09:10:06.409729 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.409866 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:10:06.411451 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.410965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58098)
0806 09:10:06.411156 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.411280 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:10:06.411731 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.469717 19325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 2
I20260806 09:10:06.471247 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.470574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58102)
0806 09:10:06.470912 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.471031 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260806 09:10:06.472741 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.472246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58104)
0806 09:10:06.472428 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.472530 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:10:06.473026 19325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.479995 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 2
I20260806 09:10:06.481407 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.480792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58114)
0806 09:10:06.481054 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.481179 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:10:06.482710 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.482259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58130)
0806 09:10:06.482435 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.482531 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:10:06.483003 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.564313 19325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 3
I20260806 09:10:06.565842 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.565113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58142)
0806 09:10:06.565413 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.565561 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 09:10:06.567425 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.566833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58156)
0806 09:10:06.567131 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.567242 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260806 09:10:06.567754 19325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.589289 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 3
I20260806 09:10:06.590751 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.590055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58168)
0806 09:10:06.590339 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.590467 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 09:10:06.592221 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.591773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58174)
0806 09:10:06.591952 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.592074 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:10:06.592515 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.711002 19325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 4
I20260806 09:10:06.712471 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.711869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58190)
0806 09:10:06.712198 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.712298 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 09:10:06.713667 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.713289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58194)
0806 09:10:06.713478 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.713550 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 09:10:06.713938 19325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
W20260806 09:10:06.761453 19334 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:10:06.761876 19334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:06.761981 19334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:06.762156 19334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:10:06.769838 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 4
W20260806 09:10:06.770078 19334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:06.770212 19334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:06.770336 19334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:06.770430 19334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:06.770534 19334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:06.771368 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.770804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58198)
0806 09:10:06.771042 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.771168 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:10:06.772773 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.772346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58208)
0806 09:10:06.772519 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.772615 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:10:06.773092 19250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.187.188:37163: Network error: Client connection negotiation failed: client connection to 127.27.187.188:37163: connect: Connection refused (error 111)
I20260806 09:10:06.783301 19334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.187.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.187.190:42381,127.27.187.189:41331,127.27.187.188:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/krb5kdc/kudu_127.27.187.188.keytab
--principal=kudu/127.27.187.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.187.188:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.187.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:06.785080 19334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:06.786665 19334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:06.799893 19339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:06.799932 19342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:06.799906 19340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:06.800966 19334 server_base.cc:1034] running on GCE node
I20260806 09:10:06.801865 19334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:06.803593 19334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:06.804884 19334 hybrid_clock.cc:648] HybridClock initialized: now 1786007406804790 us; error 63 us; skew 500 ppm
I20260806 09:10:06.805295 19334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:06.809073 19334 init.cc:377] Logged in from keytab as kudu/127.27.187.188@KRBTEST.COM (short username kudu)
I20260806 09:10:06.810986 19334 webserver.cc:460] Webserver started at http://127.27.187.188:46465/ using document root <none> and password file <none>
I20260806 09:10:06.811803 19334 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:06.811952 19334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:06.812321 19334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:06.815631 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data/instance:
uuid: "fc3473564da3460abec167b5a2f44425"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "2063a888a09a9af89e466fc725e09140"
server_key_iv: "011a79a96c825c1cdf1b5c4bfcf536e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.816615 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal/instance:
uuid: "fc3473564da3460abec167b5a2f44425"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "2063a888a09a9af89e466fc725e09140"
server_key_iv: "011a79a96c825c1cdf1b5c4bfcf536e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.823800 19334 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260806 09:10:06.828523 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:06.830276 19334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:10:06.830499 19334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data,/tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal
uuid: "fc3473564da3460abec167b5a2f44425"
format_stamp: "Formatted at 2026-08-06 09:10:06 on dist-test-slave-8tdl"
server_key: "2063a888a09a9af89e466fc725e09140"
server_key_iv: "011a79a96c825c1cdf1b5c4bfcf536e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:06.830797 19334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:06.844986 19334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:06.849864 19334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:06.850175 19334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:06.871189 19334 rpc_server.cc:307] RPC server started. Bound to: 127.27.187.188:37163
I20260806 09:10:06.871246 19400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.187.188:37163 every 8 connection(s)
I20260806 09:10:06.872778 19334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/data/info.pb
I20260806 09:10:06.877461 19402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:06.877825 28398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu as pid 19334
I20260806 09:10:06.878023 28398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvYHTS_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007003447047-28398-0/minicluster-data/master-2/wal/instance
I20260806 09:10:06.878566 28398 external_mini_cluster.cc:1428] Setting key 0a4982a28ab0b0d2b46c45ed0fcabb6a
I20260806 09:10:06.888425 19402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:06.905491 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.880529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.27.187.188:37163)
0806 09:10:06.881454 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.881467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.881511 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:10:06.883221 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.883249 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.883754 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.884161 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.886236 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.887693 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.889703 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.890548 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.890785 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.896279 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.896288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.896299 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.896340 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.900116 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.900999 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.901009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.901020 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.901285 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.901824 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.901828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.901830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.902513 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.902909 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.903365 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.903876 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":438,"threads_started":1}
I20260806 09:10:06.911026 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.891801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.27.187.190:42381)
0806 09:10:06.892478 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.892486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.892863 (+ 377us) server_negotiation.cc:407] Connection header received
0806 09:10:06.893091 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.893095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.893164 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.893282 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.894536 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.895485 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.897257 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.898007 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.898167 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.902443 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.902452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.902457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.902512 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.907380 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.908213 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.908218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.908220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.908364 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.909311 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.909316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.909318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.909632 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.909815 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.910323 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.910565 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":257,"threads_started":1}
I20260806 09:10:06.911038 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.891820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.190:42381 (local address 127.0.0.1:43628)
0806 09:10:06.892731 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.892768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.892953 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.893357 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.893366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.894037 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.894338 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.894347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.895660 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.895666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.897077 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.897095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.898319 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.898328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.898536 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.899278 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.899295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.902198 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.907540 (+ 5342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.907557 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.907568 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.908038 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.908493 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.908498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.908502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.908661 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.909796 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.909805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.909997 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.910207 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.910566 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":536,"threads_started":1}
I20260806 09:10:06.912755 19402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.190@KRBTEST.COM
I20260806 09:10:06.926817 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.908886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.27.187.190:42381)
0806 09:10:06.909509 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.909514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.909533 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:10:06.909575 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.909579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.909642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.909775 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.911423 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.912445 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.914318 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.915009 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.915140 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.920247 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.920253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.920257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.920299 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.923526 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.924426 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.924432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.924436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.924635 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.925116 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.925120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.925122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.925416 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.925804 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.926080 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.926337 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":285,"threads_started":1}
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.187.189@KRBTEST.COM
I20260806 09:10:06.941058 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.913599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.189:41331 (local address 127.0.0.1:36182)
0806 09:10:06.913908 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.913933 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.914064 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.925121 (+ 11057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.925126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.925597 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.925838 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.925848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.927079 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.927087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.928310 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.928327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.929473 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.929478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.929632 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.930367 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.930398 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.933162 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.937490 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.937494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.937497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.937979 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.938615 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.938622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.938631 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.938927 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.939883 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.939891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.940162 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.940382 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.940761 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260806 09:10:06.941334 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.913684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.27.187.189:41331)
0806 09:10:06.924740 (+ 11056us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.924745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.924763 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:10:06.924830 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.924834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.924912 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.925052 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.925999 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.926853 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.928500 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.929299 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.929436 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.933380 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.933385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.933389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.933428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.937302 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.938208 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.938214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.938217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.938419 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.939067 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.939070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.939072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.939640 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.939800 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.940572 (+ 772us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.940854 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10938,"thread_start_us":10806,"threads_started":1}
I20260806 09:10:06.942483 19402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:10:06.948413 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.928758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.27.187.189:41331)
0806 09:10:06.929410 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.929415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.929431 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:10:06.929476 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.929481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.929575 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.929678 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.931572 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.932497 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.934028 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.934724 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.934879 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.940904 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.940910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.940914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.940965 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.944718 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.945586 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.945592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.945595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.945885 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.946409 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.946413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.946415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.946834 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.947067 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.947487 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.947864 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":301,"threads_started":1}
Aug 06 09:10:06 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.188@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:06.955600 28398 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:10:06.960817 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.943493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58220)
0806 09:10:06.943694 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:06.943715 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:06.943835 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:06.944151 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:06.944154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:06.944587 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:06.944727 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.944735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.946065 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.946070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.947453 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:06.947469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.948601 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.948610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:06.948827 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.949573 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:06.949593 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:06.953054 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:06.957789 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.957794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.957797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.958281 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.958829 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:06.958833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:06.958836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:06.959045 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:06.959673 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:06.959677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:06.959830 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 09:10:06.959984 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:10:06.960376 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 09:10:06.960875 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:06.943535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.27.187.188:37163)
0806 09:10:06.943694 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:06.943700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:06.943772 (+ 72us) server_negotiation.cc:407] Connection header received
0806 09:10:06.943959 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:06.943964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:06.944035 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:06.944132 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:06.944934 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.945913 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.947707 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:06.948351 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:06.948510 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:06.953390 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:06.953399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:06.953405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:06.953465 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:06.957568 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.958437 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.958443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.958450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.958615 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:06.959146 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:06.959153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:06.959158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:06.959557 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:06.959717 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:06.960056 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 09:10:06.960551 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":60}
I20260806 09:10:06.959498 19225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e3d75c4e03fa407799ddd8c66672002e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43636
I20260806 09:10:06.966741 19402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: Bootstrap starting.
I20260806 09:10:06.970418 19402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:06.971803 19402 log.cc:824] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:06.975600 19402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: No bootstrap required, opened a new log
I20260806 09:10:06.984179 19402 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:06.984833 19402 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3473564da3460abec167b5a2f44425, State: Initialized, Role: FOLLOWER
I20260806 09:10:06.985555 19402 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:06.986693 19412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:06.987121 19412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:06.987874 19402 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:10:06.998502 19423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:10:06.998700 19423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:10:06.996479 19334 master_runner.cc:186] Error getting master registration for 127.27.187.190:42381: OK, instance_id { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" instance_seqno: 1786007405615419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:10:07.009912 19334 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:10:07.013154 19325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 5
Aug 06 09:10:07 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.189@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:07.033156 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.014017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58234)
0806 09:10:07.014279 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:07.014305 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:07.014448 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:07.014922 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:07.014930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:07.015566 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:07.016032 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:07.016055 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.018554 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.018561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:07.020098 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:07.020118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.021358 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.021365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:07.021651 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.022465 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:07.022484 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:07.025074 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:07.029999 (+ 4925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:07.030006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:07.030010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:07.030601 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:07.031116 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:07.031121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:07.031125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:07.031334 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:07.032061 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:07.032065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:07.032232 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 09:10:07.032377 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 09:10:07.032730 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 09:10:07.033169 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.014245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.27.187.188:37163)
0806 09:10:07.014506 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.014513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.014538 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:10:07.014612 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.014618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.014746 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.014864 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.016532 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.018307 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.020335 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.021135 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.021310 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.025329 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:07.025339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:07.025345 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:07.025400 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:07.029772 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.030807 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.030815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.030819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.030981 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.031483 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.031489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.031493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.031922 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:07.032130 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:07.032542 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 09:10:07.032801 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:10:07.038569 19325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: Bootstrap starting.
I20260806 09:10:07.043026 19325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:07.044225 19325 log.cc:824] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:07.047400 19325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: No bootstrap required, opened a new log
I20260806 09:10:07.050359 19250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } attempt: 5
I20260806 09:10:07.059429 19325 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:07.059944 19325 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28cbfd7f2db4cf89e7b1bf95bbee842, State: Initialized, Role: FOLLOWER
I20260806 09:10:07.060570 19325 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
Aug 06 09:10:07 dist-test-slave-8tdl krb5kdc[19169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007405, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.187.190@KRBTEST.COM for kudu/127.27.187.188@KRBTEST.COM
I20260806 09:10:07.061764 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.062275 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.063400 19325 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:07.070817 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.051115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.187.188:37163 (local address 127.0.0.1:58240)
0806 09:10:07.051360 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:07.051385 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:07.051535 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:07.052210 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:07.052216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:07.052783 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:07.053024 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:07.053040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.054729 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.054736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:07.056325 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:07.056346 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.057608 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.057616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:07.057866 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.058747 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:07.058778 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:07.062155 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:07.066995 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:07.067002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:07.067006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:07.067622 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:07.068200 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:07.068206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:07.068209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:07.068474 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:07.069451 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:07.069460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:07.069788 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 09:10:07.069958 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 09:10:07.070325 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 09:10:07.072472 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.051543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.27.187.188:37163)
0806 09:10:07.051814 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.051822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.051842 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:10:07.051906 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.051912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.051993 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.052122 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.053243 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.054474 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.056605 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.057382 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.060752 (+ 3370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.062453 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:07.062461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:07.062467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:07.062509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:07.066750 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.067831 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.067838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.067842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.068035 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.068683 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.068692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.068698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.069163 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:07.069367 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:07.071680 (+ 2313us) server_negotiation.cc:299] Negotiation successful
0806 09:10:07.072055 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 09:10:07.079154 19438 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:10:07.080806 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: Bootstrap starting.
I20260806 09:10:07.085371 19250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:07.087045 19250 log.cc:824] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:07.091734 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: No bootstrap required, opened a new log
I20260806 09:10:07.081879 19257 master_runner.cc:186] Error getting master registration for 127.27.187.190:42381: OK, instance_id { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" instance_seqno: 1786007405615419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:10:07.092945 19257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:10:07.098011 19250 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:07.098435 19250 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d75c4e03fa407799ddd8c66672002e, State: Initialized, Role: FOLLOWER
I20260806 09:10:07.099037 19250 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:07.100322 19440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.100943 19440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.101841 19250 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:07.102002 19225 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:10:07.102254 19225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:10:07.102662 19355 tablet.cc:2369] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:10:07.103353 19203 tablet.cc:2369] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:07.108028 19225 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:07.111110 19225 leader_election.cc:288] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [CANDIDATE]: Term 1 election: Requested vote from peers b28cbfd7f2db4cf89e7b1bf95bbee842 (127.27.187.189:41331), fc3473564da3460abec167b5a2f44425 (127.27.187.188:37163)
I20260806 09:10:07.111431 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d75c4e03fa407799ddd8c66672002e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842"
I20260806 09:10:07.111974 19300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:07.115653 19300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d75c4e03fa407799ddd8c66672002e in term 1.
I20260806 09:10:07.116712 19376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3d75c4e03fa407799ddd8c66672002e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc3473564da3460abec167b5a2f44425"
I20260806 09:10:07.117264 19376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:07.119053 19201 leader_election.cc:302] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b28cbfd7f2db4cf89e7b1bf95bbee842, e3d75c4e03fa407799ddd8c66672002e; no voters:
I20260806 09:10:07.119717 19440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:07.123342 19376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d75c4e03fa407799ddd8c66672002e in term 1.
I20260806 09:10:07.127084 19440 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [term 1 LEADER]: Becoming Leader. State: Replica: e3d75c4e03fa407799ddd8c66672002e, State: Running, Role: LEADER
I20260806 09:10:07.128170 19440 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } }
I20260806 09:10:07.148397 19452 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d75c4e03fa407799ddd8c66672002e. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.148901 19452 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:07.150163 19459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:07.170436 19459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:07.180413 19376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [term 1 FOLLOWER]: Refusing update from remote peer e3d75c4e03fa407799ddd8c66672002e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:07.181836 19452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:07.183501 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.130624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.27.187.190:42381)
0806 09:10:07.131095 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.131101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.141150 (+ 10049us) server_negotiation.cc:407] Connection header received
0806 09:10:07.141726 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.141733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.141861 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.141939 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.144697 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.145869 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.154357 (+ 8488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.155706 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.155905 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.161677 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:07.161686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:07.161690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:07.161741 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:07.167164 (+ 5423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.180439 (+ 13275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.180450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.180457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.180706 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.181315 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.181323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.181330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.181779 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:07.182076 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:07.182777 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 09:10:07.183095 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260806 09:10:07.194561 19300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [term 1 FOLLOWER]: Refusing update from remote peer e3d75c4e03fa407799ddd8c66672002e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:07.196055 19452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:07.224498 19412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d75c4e03fa407799ddd8c66672002e. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.224961 19412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.224980 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.158931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.27.187.188:37163)
0806 09:10:07.159201 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.159212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.184666 (+ 25454us) server_negotiation.cc:407] Connection header received
0806 09:10:07.184912 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.184921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.185038 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.185159 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.187768 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.188968 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.203724 (+ 14756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.204562 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.204704 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.210567 (+ 5863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:07.210579 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:07.210589 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:07.210667 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:07.217093 (+ 6426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.220421 (+ 3328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.220433 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.220441 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.220726 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.221453 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.221463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.221470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.222118 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:07.222516 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:07.223927 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0806 09:10:07.224310 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:10:07.231813 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:07.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.27.187.189:41331)
0806 09:10:07.131043 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.131050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.156945 (+ 25895us) server_negotiation.cc:407] Connection header received
0806 09:10:07.157198 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.157211 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.157323 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.157436 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.176768 (+ 19332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.177918 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.185018 (+ 7100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:07.185879 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:07.186028 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:07.190521 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:07.190532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:07.190539 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:07.190595 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:07.201617 (+ 11022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.228129 (+ 26512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.228136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.228139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.228358 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:07.228945 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:07.228951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:07.228956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:07.229410 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:07.229701 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:07.230845 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0806 09:10:07.231263 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 09:10:07.239120 19412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.239475 19412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.243583 19440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.244082 19440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:07.245391 19459 catalog_manager.cc:1347] Generated new cluster ID: 2c91671666444f21b6580f36eb1ebc99
I20260806 09:10:07.245517 19459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:07.247484 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d75c4e03fa407799ddd8c66672002e. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.247869 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.250324 19452 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.250665 19452 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e [sys.catalog]: This master's current role is: LEADER
W20260806 09:10:07.258174 19279 tablet.cc:2369] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:07.261088 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3d75c4e03fa407799ddd8c66672002e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d75c4e03fa407799ddd8c66672002e" member_type: VOTER last_known_addr { host: "127.27.187.190" port: 42381 } } peers { permanent_uuid: "b28cbfd7f2db4cf89e7b1bf95bbee842" member_type: VOTER last_known_addr { host: "127.27.187.189" port: 41331 } } peers { permanent_uuid: "fc3473564da3460abec167b5a2f44425" member_type: VOTER last_known_addr { host: "127.27.187.188" port: 37163 } } }
I20260806 09:10:07.261402 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:10:07.276753 19459 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:07.279078 19459 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:07.304916 19459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3d75c4e03fa407799ddd8c66672002e: Generated new TSK 0
I20260806 09:10:07.306273 19459 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:10:07.420581 19408 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.409311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.27.187.190:42381)
0806 09:10:07.409494 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.409500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.416280 (+ 6780us) server_negotiation.cc:407] Connection header received
0806 09:10:07.418306 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.418315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.418433 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.418526 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.420177 (+ 1651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43656: BlockingRecv error: recv got EOF from 127.0.0.1:43656 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:10:07.422559 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.408936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.27.187.189:41331)
0806 09:10:07.409195 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.409203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.419346 (+ 10143us) server_negotiation.cc:407] Connection header received
0806 09:10:07.419898 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.419905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.420003 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.420079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.422016 (+ 1937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36208: BlockingRecv error: recv got EOF from 127.0.0.1:36208 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 09:10:07.425859 19405 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.417064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.27.187.188:37163)
0806 09:10:07.417311 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.417318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.422985 (+ 5667us) server_negotiation.cc:407] Connection header received
0806 09:10:07.423824 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.423830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.423932 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.424010 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.425558 (+ 1548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58262: BlockingRecv error: recv got EOF from 127.0.0.1:58262 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 09:10:07.433993 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.431268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.27.187.189:41331)
0806 09:10:07.431485 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.431493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.432126 (+ 633us) server_negotiation.cc:407] Connection header received
0806 09:10:07.432425 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.432433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.432618 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.432739 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.433748 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36216: BlockingRecv error: recv got EOF from 127.0.0.1:36216 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 09:10:07.441769 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.439090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.27.187.189:41331)
0806 09:10:07.439330 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.439338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.439944 (+ 606us) server_negotiation.cc:407] Connection header received
0806 09:10:07.440264 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.440270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.440378 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.440494 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.441478 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36232: BlockingRecv error: recv got EOF from 127.0.0.1:36232 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260806 09:10:07.451771 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.448915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.27.187.189:41331)
0806 09:10:07.449146 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.449154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.449907 (+ 753us) server_negotiation.cc:407] Connection header received
0806 09:10:07.450023 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.450030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.450159 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.450303 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.451460 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36234: BlockingRecv error: recv got EOF from 127.0.0.1:36234 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:10:07.465243 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.461165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.27.187.189:41331)
0806 09:10:07.461399 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.461407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.462275 (+ 868us) server_negotiation.cc:407] Connection header received
0806 09:10:07.462512 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.462520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.462651 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.462811 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.464974 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36248: BlockingRecv error: recv got EOF from 127.0.0.1:36248 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 09:10:07.479807 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.476719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.27.187.189:41331)
0806 09:10:07.477173 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.477180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.477929 (+ 749us) server_negotiation.cc:407] Connection header received
0806 09:10:07.478134 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.478140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.478279 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.478411 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.479483 (+ 1072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36262: BlockingRecv error: recv got EOF from 127.0.0.1:36262 (error 108)
Metrics: {"server-negotiator.queue_time_us":301}
W20260806 09:10:07.502470 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.499782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.27.187.189:41331)
0806 09:10:07.500004 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.500011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.500659 (+ 648us) server_negotiation.cc:407] Connection header received
0806 09:10:07.500890 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.500897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.501024 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.501171 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.502163 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36278: BlockingRecv error: recv got EOF from 127.0.0.1:36278 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:10:07.543025 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.540461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.27.187.189:41331)
0806 09:10:07.540674 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.540690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.541256 (+ 566us) server_negotiation.cc:407] Connection header received
0806 09:10:07.541506 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.541515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.541635 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.541764 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.542784 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36280: BlockingRecv error: recv got EOF from 127.0.0.1:36280 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 09:10:07.613692 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.611283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.27.187.189:41331)
0806 09:10:07.611468 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.611473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.611824 (+ 351us) server_negotiation.cc:407] Connection header received
0806 09:10:07.612096 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.612102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.612212 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.612309 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.613413 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36284: BlockingRecv error: recv got EOF from 127.0.0.1:36284 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 09:10:07.748188 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:07.746051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.27.187.189:41331)
0806 09:10:07.746287 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:07.746293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:07.746623 (+ 330us) server_negotiation.cc:407] Connection header received
0806 09:10:07.746896 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:07.746901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:07.746988 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:07.747093 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:07.747984 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36298: BlockingRecv error: recv got EOF from 127.0.0.1:36298 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:10:08.001041 19423 catalog_manager.cc:1259] Loaded cluster ID: 2c91671666444f21b6580f36eb1ebc99
I20260806 09:10:08.001174 19423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: loading cluster ID for follower catalog manager: success
I20260806 09:10:08.004349 19423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: acquiring CA information for follower catalog manager: success
I20260806 09:10:08.008423 19423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc3473564da3460abec167b5a2f44425: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:10:08.011791 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:08.009702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.27.187.189:41331)
0806 09:10:08.009962 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.009969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.010133 (+ 164us) server_negotiation.cc:407] Connection header received
0806 09:10:08.010396 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.010401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.010499 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.010620 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:08.011523 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36308: BlockingRecv error: recv got EOF from 127.0.0.1:36308 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:10:08.084210 19438 catalog_manager.cc:1259] Loaded cluster ID: 2c91671666444f21b6580f36eb1ebc99
I20260806 09:10:08.084432 19438 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: loading cluster ID for follower catalog manager: success
I20260806 09:10:08.087378 19438 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: acquiring CA information for follower catalog manager: success
I20260806 09:10:08.089743 19438 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b28cbfd7f2db4cf89e7b1bf95bbee842: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:10:08.539742 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:08.532458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.27.187.189:41331)
0806 09:10:08.533347 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.533352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.533375 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:10:08.533565 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.533569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.533750 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.533869 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:08.534352 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.535432 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.537098 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.537576 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.537746 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:08.538251 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:08.538929 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:08.539058 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:08.539342 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 09:10:08.539438 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":666,"threads_started":1}
I20260806 09:10:08.554081 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:08.546814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.27.187.188:37163)
0806 09:10:08.547734 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.547738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.547756 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:10:08.547857 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.547861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.548034 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.548203 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:08.548674 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.549507 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.551206 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.551667 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.552112 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:08.552551 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:08.553270 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:08.553398 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:08.553598 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 09:10:08.553689 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":636,"threads_started":1}
I20260806 09:10:08.570632 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:08.561118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.27.187.190:42381)
0806 09:10:08.563011 (+ 1893us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.563020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.563057 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:10:08.563204 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.563210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.563455 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.563670 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:08.564450 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.565569 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.567764 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.568474 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.568629 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:08.569330 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:08.569829 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:08.570002 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:08.570161 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 09:10:08.570277 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1742,"thread_start_us":1629,"threads_started":1}
I20260806 09:10:08.570745 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:08.562311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.27.187.188:37163)
0806 09:10:08.562520 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.562527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.563429 (+ 902us) server_negotiation.cc:407] Connection header received
0806 09:10:08.563606 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.563611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.563705 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.563803 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:08.564612 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.565400 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.568175 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.568960 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.569091 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:08.569762 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:08.569947 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:08.570032 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:08.570309 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 09:10:08.570424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:10:08.571398 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:08.561181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.27.187.189:41331)
0806 09:10:08.561410 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:08.561420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:08.562801 (+ 1381us) server_negotiation.cc:407] Connection header received
0806 09:10:08.563171 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:08.563178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:08.563264 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:08.563317 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:08.564250 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.565443 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.568794 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:08.569573 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:08.569728 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:08.570393 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:08.570578 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:08.570652 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:08.570986 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 09:10:08.571119 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:10:08.576444 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 19182
I20260806 09:10:08.577617 19245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:08.694881 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 19182
W20260806 09:10:08.719662 19274 connection.cc:535] server connection from 127.0.0.1:36176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:10:08.720507 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 19257
I20260806 09:10:08.721858 19320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:08.840945 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 19257
I20260806 09:10:08.866281 28398 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu pid 19334
I20260806 09:10:08.867914 19396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:08.982214 28398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvYHTS_/build/asan/bin/kudu with pid 19334
2026-08-06T09:10:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5456 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10594 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (405426 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST